scripts/callcenter.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "callcenter.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mQt1tRpl2qC7",
        "colab_type": "text"
      },
      "source": [
        "Этот файлик содержит решение четвёртой задачки - про колл-центр. \n",
        "\n",
        "Решение не очень интересное: фича-инжиниринг и подбор параметров катбуста. Удивительно, что предложенный нам бейзлайн вообще использовал очевидно более слабые линейные классификаторы. \n",
        "\n",
        "Фичи: \n",
        "* все моменты времени - в секундах от полуночи, и синус-косинус часовой/минутной/секундной стрелки. \n",
        "* индикаторы пропусков всех признаков\n",
        "* время между всеми событиями\n",
        "* доли каждого этапа (гудки, постановка в очередь, разговор с оператором) от всего времени звонка. \n",
        "* разница проставленного времени звонка разговора с оператором и рассчитанного по началу-концу\n",
        "* пробовал ещё разные линейные функции от этих фич, но они не принесли профита \n",
        "\n",
        "Удивительно, но абсолютное время в очереди (достаточно большое) и время суток (глубокая ночь) оказались суперфичами, дающими большую часть предсказательной силы. Этому есть некоторое логическое объяснение: чем дольше звонок в очереди, тем стыднее (и тем больше стимулов врать), а ночью операторов меньше всего, и это тоже, видимо, увеличивает время ожидания. Тем не менее, у меня были большие опасения, что мы переобучимся на эти фичи, а в тестовом сете зависимость от них будет другой. Но вроде бы этого не произошло. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ld5bv5V72y9q",
        "colab_type": "text"
      },
      "source": [
        "## подготовка данных"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tEiUE8F4Z9qB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!wget https://github.com/datasouls/nlp-hack-2019/raw/master/data/callcenter_train.csv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H00x_E2Gg-3Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install catboost"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "znPhQPugaJXI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WIyP1QhFaK-F",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "all_data = pd.read_csv('callcenter_train.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oFejJlnJaPC4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "51532bee-00ed-4905-e1f9-9b8be3a0884f"
      },
      "source": [
        "all_data.shape"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(30804, 8)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J9Pvskv1aaUs",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 686
        },
        "outputId": "348e2ec9-f2f5-4b12-a0a4-c40ae44f46ac"
      },
      "source": [
        "all_data.sample(20).sort_values(\"Метка\")\n"
      ],
      "execution_count": 530,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>Время начала вызова</th>\n",
              "      <th>Время окончания вызова</th>\n",
              "      <th>Время постановки в очередь</th>\n",
              "      <th>Время переключения на оператора</th>\n",
              "      <th>Время окончания разговора с оператором</th>\n",
              "      <th>Длительность разговора с оператором, сек</th>\n",
              "      <th>Метка</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>23688</th>\n",
              "      <td>23688</td>\n",
              "      <td>12:59:56</td>\n",
              "      <td>13:01:20</td>\n",
              "      <td>12:59:56</td>\n",
              "      <td>12:59:56</td>\n",
              "      <td>13:01:17</td>\n",
              "      <td>81.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21685</th>\n",
              "      <td>21685</td>\n",
              "      <td>12:00:05</td>\n",
              "      <td>12:02:58</td>\n",
              "      <td>12:01:13</td>\n",
              "      <td>12:01:13</td>\n",
              "      <td>12:02:54</td>\n",
              "      <td>101.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10594</th>\n",
              "      <td>10594</td>\n",
              "      <td>19:51:40</td>\n",
              "      <td>19:54:28</td>\n",
              "      <td>19:52:48</td>\n",
              "      <td>19:52:48</td>\n",
              "      <td>19:53:49</td>\n",
              "      <td>61.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18613</th>\n",
              "      <td>18613</td>\n",
              "      <td>21:02:33</td>\n",
              "      <td>21:02:43</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7658</th>\n",
              "      <td>7658</td>\n",
              "      <td>16:28:52</td>\n",
              "      <td>16:33:40</td>\n",
              "      <td>16:29:56</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26725</th>\n",
              "      <td>26725</td>\n",
              "      <td>21:11:41</td>\n",
              "      <td>21:14:41</td>\n",
              "      <td>21:12:21</td>\n",
              "      <td>21:12:21</td>\n",
              "      <td>21:14:40</td>\n",
              "      <td>139.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8995</th>\n",
              "      <td>8995</td>\n",
              "      <td>10:20:43</td>\n",
              "      <td>10:23:33</td>\n",
              "      <td>10:21:16</td>\n",
              "      <td>10:22:09</td>\n",
              "      <td>10:23:33</td>\n",
              "      <td>84.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26084</th>\n",
              "      <td>26084</td>\n",
              "      <td>15:48:53</td>\n",
              "      <td>15:49:39</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28069</th>\n",
              "      <td>28069</td>\n",
              "      <td>14:14:59</td>\n",
              "      <td>14:23:18</td>\n",
              "      <td>14:16:07</td>\n",
              "      <td>14:16:16</td>\n",
              "      <td>14:22:52</td>\n",
              "      <td>396.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27009</th>\n",
              "      <td>27009</td>\n",
              "      <td>04:18:45</td>\n",
              "      <td>04:19:55</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20548</th>\n",
              "      <td>20548</td>\n",
              "      <td>18:26:29</td>\n",
              "      <td>18:27:29</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19086</th>\n",
              "      <td>19086</td>\n",
              "      <td>08:59:11</td>\n",
              "      <td>08:59:35</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16622</th>\n",
              "      <td>16622</td>\n",
              "      <td>16:18:19</td>\n",
              "      <td>16:22:01</td>\n",
              "      <td>16:19:24</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29944</th>\n",
              "      <td>29944</td>\n",
              "      <td>13:24:25</td>\n",
              "      <td>13:31:46</td>\n",
              "      <td>13:25:31</td>\n",
              "      <td>13:27:02</td>\n",
              "      <td>13:31:14</td>\n",
              "      <td>252.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27478</th>\n",
              "      <td>27478</td>\n",
              "      <td>10:39:26</td>\n",
              "      <td>10:48:28</td>\n",
              "      <td>10:40:34</td>\n",
              "      <td>10:40:48</td>\n",
              "      <td>10:47:50</td>\n",
              "      <td>422.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>825</th>\n",
              "      <td>825</td>\n",
              "      <td>13:09:16</td>\n",
              "      <td>13:14:30</td>\n",
              "      <td>13:10:22</td>\n",
              "      <td>13:10:42</td>\n",
              "      <td>13:14:08</td>\n",
              "      <td>206.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>239</th>\n",
              "      <td>239</td>\n",
              "      <td>09:00:48</td>\n",
              "      <td>09:02:36</td>\n",
              "      <td>9:01:56</td>\n",
              "      <td>09:02:27</td>\n",
              "      <td>09:02:36</td>\n",
              "      <td>9.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29875</th>\n",
              "      <td>29875</td>\n",
              "      <td>12:47:13</td>\n",
              "      <td>12:51:31</td>\n",
              "      <td>12:48:23</td>\n",
              "      <td>12:48:36</td>\n",
              "      <td>12:51:30</td>\n",
              "      <td>174.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25573</th>\n",
              "      <td>25573</td>\n",
              "      <td>12:03:28</td>\n",
              "      <td>12:07:00</td>\n",
              "      <td>12:04:39</td>\n",
              "      <td>12:05:08</td>\n",
              "      <td>12:06:57</td>\n",
              "      <td>109.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5725</th>\n",
              "      <td>5725</td>\n",
              "      <td>12:02:44</td>\n",
              "      <td>12:09:13</td>\n",
              "      <td>12:03:51</td>\n",
              "      <td>12:08:01</td>\n",
              "      <td>12:08:46</td>\n",
              "      <td>45.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "          id  ... Метка\n",
              "23688  23688  ...     0\n",
              "21685  21685  ...     0\n",
              "10594  10594  ...     0\n",
              "18613  18613  ...     0\n",
              "7658    7658  ...     0\n",
              "26725  26725  ...     0\n",
              "8995    8995  ...     0\n",
              "26084  26084  ...     0\n",
              "28069  28069  ...     0\n",
              "27009  27009  ...     0\n",
              "20548  20548  ...     0\n",
              "19086  19086  ...     0\n",
              "16622  16622  ...     0\n",
              "29944  29944  ...     1\n",
              "27478  27478  ...     1\n",
              "825      825  ...     1\n",
              "239      239  ...     1\n",
              "29875  29875  ...     1\n",
              "25573  25573  ...     1\n",
              "5725    5725  ...     1\n",
              "\n",
              "[20 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 530
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0_3jDQZ0MjPl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZDq2M8iNE3wy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        },
        "outputId": "914bc6c2-7fc6-495f-e61d-225ea349db86"
      },
      "source": [
        "all_data.isnull().mean()"
      ],
      "execution_count": 453,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "id                                          0.000000\n",
              "Время начала вызова                         0.000000\n",
              "Время окончания вызова                      0.000032\n",
              "Время постановки в очередь                  0.278308\n",
              "Время переключения на оператора             0.395111\n",
              "Время окончания разговора с оператором      0.395403\n",
              "Длительность разговора с оператором, сек    0.395079\n",
              "Метка                                       0.000000\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 453
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RD0e9MBfaYdt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import train_test_split"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NB3FThI6afSw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "all_train, all_test0 = train_test_split(all_data, random_state=1, test_size=0.3)\n",
        "all_test, all_valid = train_test_split(all_test0, random_state=1, test_size=0.3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YLA1vQl323S_",
        "colab_type": "text"
      },
      "source": [
        "## тренируемся работать со временем"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cnPBdB2cbTnA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "93952552-9f78-4650-ad3a-3ac2ff16602f"
      },
      "source": [
        "all_train['Время начала вызова'].iloc[0]"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'09:48:36'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aaLjH_r2bbEF",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "241d247d-4853-4516-fd1d-d777fab18573"
      },
      "source": [
        "pd.to_datetime(all_train['Время постановки в очередь'].iloc[0:10])"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "27326                   NaT\n",
              "1114    2019-12-15 15:18:16\n",
              "8517    2019-12-15 23:01:54\n",
              "28473   2019-12-15 16:38:32\n",
              "26942   2019-12-15 01:28:23\n",
              "25674   2019-12-15 12:47:02\n",
              "19875   2019-12-15 14:09:09\n",
              "29505                   NaT\n",
              "20217   2019-12-15 16:28:23\n",
              "1256    2019-12-15 16:19:52\n",
              "Name: Время постановки в очередь, dtype: datetime64[ns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o4JSBH37b_CX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "9e4e3a2f-e5ce-4930-b009-b2fdf7400b8f"
      },
      "source": [
        "\n",
        "all_data.columns"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['id', 'Время начала вызова', 'Время окончания вызова',\n",
              "       'Время постановки в очередь', 'Время переключения на оператора',\n",
              "       'Время окончания разговора с оператором',\n",
              "       'Длительность разговора с оператором, сек', 'Метка'],\n",
              "      dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p3ncvFx8dMxU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "7ebb6b45-5f0a-48c7-885c-4b41d9d97d1a"
      },
      "source": [
        "\n",
        "pd.to_datetime(all_train['Время окончания разговора с оператором'])"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "27326                   NaT\n",
              "1114    2019-12-15 15:19:33\n",
              "8517    2019-12-15 23:07:53\n",
              "28473   2019-12-15 16:41:34\n",
              "26942                   NaT\n",
              "                ...        \n",
              "17289   2019-12-15 05:17:10\n",
              "5192    2019-12-15 10:39:11\n",
              "12172   2019-12-15 15:11:38\n",
              "235                     NaT\n",
              "29733                   NaT\n",
              "Name: Время окончания разговора с оператором, Length: 21562, dtype: datetime64[ns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A7DgnriM29A8",
        "colab_type": "text"
      },
      "source": [
        "## Собственно, извлечение признаков"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nX0FPPvkggrW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "DUR_RU = 'Длительность разговора с оператором, сек'\n",
        "DUR_EN = 'oper_duration'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vKArekckbFcp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "RU_COLS = ['Время начала вызова', 'Время окончания вызова', 'Время постановки в очередь', 'Время переключения на оператора', 'Время окончания разговора с оператором',]\n",
        "EN_COLS = ['call_start_time', 'call_end_time', 'queue_time', 'oper_start_time', 'oper_end_time']\n",
        "\n",
        "zero_time = pd.to_datetime('00:00:00')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "scU_lQgcivyQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "SEC_PER_DAY = 60*60*24"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yLbdy85WhUM4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "NA_VALUE = -1.2345\n",
        "\n",
        "\n",
        "def extract_features(data):\n",
        "    times = data[RU_COLS].apply(pd.to_datetime)\n",
        "    times.columns = EN_COLS\n",
        "    \n",
        "    abs_times = times.apply(lambda x: (x-zero_time).dt.total_seconds()).fillna(NA_VALUE)\n",
        "    abs_times.columns = [c+'_abs' for c in EN_COLS]\n",
        "\n",
        "    day = abs_times / SEC_PER_DAY * 2 * np.pi\n",
        "    hour = (abs_times % (24*60)) / (24*60) * 2 * np.pi\n",
        "    minute = (abs_times % 60) / 60 * 2 * np.pi\n",
        "    day_sines = np.sin(day)\n",
        "    day_cosines = np.cos(day)\n",
        "    hour_sines = np.sin(hour)\n",
        "    hour_cosines = np.cos(hour)\n",
        "    minute_sines = np.sin(minute)\n",
        "    minute_cosines = np.cos(minute)\n",
        "\n",
        "    day_sines.columns = ['day_sin__'+c for c in EN_COLS]\n",
        "    day_cosines.columns = ['day_cos__'+c for c in EN_COLS]\n",
        "    hour_sines.columns = ['hour_sin__'+c for c in EN_COLS]\n",
        "    hour_cosines.columns = ['hour_cos__'+c for c in EN_COLS]\n",
        "    minute_sines.columns = ['minute_sin__'+c for c in EN_COLS]\n",
        "    minute_cosines.columns = ['minute_cos__'+c for c in EN_COLS]\n",
        "\n",
        "    null_times = times.isnull().astype(int)\n",
        "    null_times.columns = [c+\"_miss\" for c in EN_COLS]\n",
        "\n",
        "    diffs = pd.DataFrame(index=times.index)\n",
        "    for i, c1 in enumerate(EN_COLS):\n",
        "        for j, c2 in enumerate(EN_COLS[(i+1):]):\n",
        "            diffs['delta_{}_{}'.format(c1, c2)] = (times[c2] - times[c1]).dt.total_seconds().fillna(NA_VALUE)\n",
        "    \n",
        "    deltas_base = 'delta_call_start_time_call_end_time'\n",
        "\n",
        "    for c in diffs.columns:\n",
        "        d = diffs[c] / diffs[deltas_base]\n",
        "        diffs['rel_{}'.format(c)] = d\n",
        "    \n",
        "    x = pd.concat([abs_times, day_sines, day_cosines, hour_sines, hour_cosines, minute_sines, minute_cosines, null_times, diffs], axis=1)\n",
        "    x[DUR_EN] = data[DUR_RU].fillna(NA_VALUE)\n",
        "    x[DUR_EN+'_miss'] = data[DUR_RU].isnull().astype(int)\n",
        "\n",
        "    devia = x['delta_oper_start_time_oper_end_time'] - x[DUR_EN]\n",
        "    devia[x['oper_duration_miss']==1] = 0\n",
        "    devia[x['delta_oper_start_time_oper_end_time'] < 0] = 0\n",
        "\n",
        "    x['oper_time_deviation'] = devia\n",
        "\n",
        "    return x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k3ITopb1ewZR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_x = extract_features(all_train)\n",
        "test_x = extract_features(all_test)\n",
        "valid_x = extract_features(all_valid)\n",
        "train_y = all_train['Метка']\n",
        "test_y = all_test['Метка']\n",
        "valid_y = all_valid['Метка']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AiSWK5ZKFkf1",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "626zHiTRoihl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 476
        },
        "outputId": "fbba6cdf-e43e-4855-e319-20841c65ed29"
      },
      "source": [
        "\n",
        "\n",
        "train_x.columns"
      ],
      "execution_count": 308,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['call_start_time_abs', 'call_end_time_abs', 'queue_time_abs',\n",
              "       'oper_start_time_abs', 'oper_end_time_abs', 'day_sin__call_start_time',\n",
              "       'day_sin__call_end_time', 'day_sin__queue_time',\n",
              "       'day_sin__oper_start_time', 'day_sin__oper_end_time',\n",
              "       'day_cos__call_start_time', 'day_cos__call_end_time',\n",
              "       'day_cos__queue_time', 'day_cos__oper_start_time',\n",
              "       'day_cos__oper_end_time', 'hour_sin__call_start_time',\n",
              "       'hour_sin__call_end_time', 'hour_sin__queue_time',\n",
              "       'hour_sin__oper_start_time', 'hour_sin__oper_end_time',\n",
              "       'hour_cos__call_start_time', 'hour_cos__call_end_time',\n",
              "       'hour_cos__queue_time', 'hour_cos__oper_start_time',\n",
              "       'hour_cos__oper_end_time', 'minute_sin__call_start_time',\n",
              "       'minute_sin__call_end_time', 'minute_sin__queue_time',\n",
              "       'minute_sin__oper_start_time', 'minute_sin__oper_end_time',\n",
              "       'minute_cos__call_start_time', 'minute_cos__call_end_time',\n",
              "       'minute_cos__queue_time', 'minute_cos__oper_start_time',\n",
              "       'minute_cos__oper_end_time', 'call_start_time_miss',\n",
              "       'call_end_time_miss', 'queue_time_miss', 'oper_start_time_miss',\n",
              "       'oper_end_time_miss', 'delta_call_start_time_call_end_time',\n",
              "       'delta_call_start_time_queue_time',\n",
              "       'delta_call_start_time_oper_start_time',\n",
              "       'delta_call_start_time_oper_end_time', 'delta_call_end_time_queue_time',\n",
              "       'delta_call_end_time_oper_start_time',\n",
              "       'delta_call_end_time_oper_end_time', 'delta_queue_time_oper_start_time',\n",
              "       'delta_queue_time_oper_end_time', 'delta_oper_start_time_oper_end_time',\n",
              "       'oper_duration', 'oper_duration_miss', 'oper_time_deviation'],\n",
              "      dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 308
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v3pBt0aDL129",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e419faf2-db13-4ee7-8c30-1ee157270c79"
      },
      "source": [
        "has_dlit = train_x[train_x.oper_duration_miss== 0] \n",
        "has_dlit_y = train_y[train_x.oper_duration_miss== 0]\n",
        "has_dlit.shape"
      ],
      "execution_count": 524,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(13052, 63)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 524
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l-Bx4ti2L_P-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "14fc7b86-55eb-4949-a170-bde20a6781b0"
      },
      "source": [
        "(has_dlit.oper_duration[has_dlit_y==1] % 2).hist(density=True, alpha=0.5)\n",
        "(has_dlit.oper_duration[has_dlit_y==0] % 2).hist(density=True, alpha=0.5)"
      ],
      "execution_count": 529,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f0f29873e10>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 529
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAANqklEQVR4nO3dbYxc51nG8etyNq3BYzBmm1UVB5aG\nBmEFodYjqFUJdlJahWAlH4hQIoUXybBqEVEFWALUL6UIVUhJkZAigYWjFkw7LS5FVkxbAvVgtbJb\ndhs3zQut0hDAJWBcu1uPLUxibj7MbOOYWc/ZmTnn3Ov5/6SRZvY8Z5773tm9fPaZc8aOCAEA8tpU\ndwEAgGsjqAEgOYIaAJIjqAEgOYIaAJKbKeNJZ2dnY35+fqR9L1y4oC1btky2oOTo+fo3bf1K9Lxe\ny8vLZyLidYO2lRLU8/PzWlpaGmnfTqejhYWFyRaUHD1f/6atX4me18v2v6y1jaUPAEiOoAaA5Ahq\nAEiOoAaA5AhqAEiOoAaA5AhqAEiOoAaA5Apd8GL7BUnnJV2W9HJENMssCgDwivVcmdiKiDOlVdJ3\n+vwl/eHjXy17mv/n199+W+VzAkARpVxCDgC1Ovr+eub17lKetugadUj6W9vLthdLqQQAMJCL/J+J\ntm+OiK/bvknS45IejIhjV41ZlLQoSXNzc7va7fZIBZ1b+ZZe2vTakfYdx01bq59zVbfbVaPRqG3+\nOkxbz9PWr1Rzz+f/o5Zpu26M3HOr1Vpe6/2/QkH9qh3s90rqRsRDa41pNpsx6qfnHTl0UNtXTo60\n7zh2712zndLxKWPXv2nrV6q555qWPjrePc6n560Z1EPXqG1vkbQpIs73779D0vtGqgQAKnD8+W/U\nM/Gt5TxtkTcT5yR9wvbq+A9HxKfKKQcAcLWhQR0Rz0v60QpqAQAMwJWJAJAcQQ0AyRHUAJAcQQ0A\nyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHU\nAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAc\nQQ0AyRHUAJAcQQ0AyRUOats32H7C9mNlFgQAeLX1HFG/W9KzZRUCABisUFDb3iHpZyT9abnlAACu\n5ogYPsg+JOn9krZK2hcRewaMWZS0KElzc3O72u32SAWtnDurmcsXR9p3HFtmd1Q+56put6tGo1Hb\n/HWYtp6nrV+p3p4vnDlVy7yxedvIPbdareWIaA7aNjNsZ9t7JJ2OiGXbC2sWGLFf0n5JajabsbCw\n5tBrOnLooLavnBxp33HsvveByudc1el0NOr3a6Oatp6nrV+p3p6PH9hXy7yXZveU0nORpY+3Srrb\n9guS2pLusH1w4pUAAAYaGtQR8TsRsSMi5iXdJ+kzEVHf4ScATBnOowaA5IauUV8pIjqSOqVUAgAY\niCNqAEiOoAaA5AhqAEiOoAaA5AhqAEiOoAaA5AhqAEiOoAaA5AhqAEiOoAaA5AhqAEiOoAaA5Ahq\nAEiOoAaA5AhqAEiOoAaA5AhqAEiOoAaA5AhqAEiOoAaA5AhqAEiOoAaA5AhqAEiOoAaA5AhqAEiO\noAaA5AhqAEiOoAaA5AhqAEiOoAaA5AhqAEiOoAaA5IYGte3Ntr9g+0u2n7b9u1UUBgDomSkw5pKk\nOyKia/tGSZ+1/cmIOFFybQAAFQjqiAhJ3f7DG/u3KLMoAMAr3MvhIYPsGyQtS/pBSY9ExG8NGLMo\naVGS5ubmdrXb7ZEKWjl3VjOXL4607zi2zO6ofM5V3W5XjUajtvnrMG09T1u/Ur09XzhzqpZ5Y/O2\nkXtutVrLEdEctK1QUH97sL1N0ickPRgRT601rtlsxtLS0roLlaQjhw5q+8rJkfYdx+69D1U+56pO\np6OFhYXa5q/DtPU8bf1K9fZ8/MC+Wua9dOuekXu2vWZQr+usj4j4pqSjku4cqRIAwLoVOevjdf0j\nadn+Dklvl/RPZRcGAOgpctbH6yV9qL9OvUnSxyLisXLLAgCsKnLWx5OS3lRBLQCAAbgyEQCSI6gB\nIDmCGgCSI6gBIDmCGgCSI6gBIDmCGgCSI6gBIDmCGgCSI6gBIDmCGgCSI6gBIDmCGgCSI6gBIDmC\nGgCSI6gBIDmCGgCSI6gBIDmCGgCSI6gBIDmCGgCSI6gBIDmCGgCSI6gBIDmCGgCSI6gBIDmCGgCS\nI6gBIDmCGgCSI6gBIDmCGgCSI6gBILmhQW37FttHbT9j+2nb766iMABAz0yBMS9L+s2I+KLtrZKW\nbT8eEc+UXBsAQAWOqCPixYj4Yv/+eUnPSrq57MIAAD2OiOKD7XlJxyTdHhHfumrboqRFSZqbm9vV\nbrdHKmjl3FnNXL440r7j2DK7o/I5V3W7XTUajdrmr8O09Txt/Ur19nzhzKla5o3N20buudVqLUdE\nc9C2wkFtuyHpHyT9fkT81bXGNpvNWFpaWnehknTk0EFtXzk50r7j2L33ocrnXNXpdLSwsFDb/HWY\ntp6nrV+p3p6PH9hXy7yXbt0zcs+21wzqQmd92L5R0scl/cWwkAYATFaRsz4s6YCkZyPiA+WXBAC4\nUpEj6rdK+nlJd9g+2b/dVXJdAIC+oafnRcRnJbmCWgAAA3BlIgAkR1ADQHIENQAkR1ADQHIENQAk\nR1ADQHIENQAkR1ADQHIENQAkR1ADQHIENQAkR1ADQHIENQAkR1ADQHIENQAkR1ADQHIENQAkR1AD\nQHIENQAkR1ADQHIENQAkR1ADQHIENQAkR1ADQHIENQAkR1ADQHIENQAkR1ADQHIENQAkR1ADQHIE\nNQAkR1ADQHJDg9r2o7ZP236qioIAAK9W5Ij6g5LuLLkOAMAahgZ1RByTdLaCWgAAAzgihg+y5yU9\nFhG3X2PMoqRFSZqbm9vVbrdHKmjl3FnNXL440r7j2DK7o/I5V3W7XTUajdrmr8O09Txt/Ur19nzh\nzKla5o3N20buudVqLUdEc9C2mbGqukJE7Je0X5KazWYsLCyM9DxHDh3U9pWTkyqrsN33PlD5nKs6\nnY5G/X5tVNPW87T1K9Xb8/ED+2qZ99LsnlJ65qwPAEiOoAaA5IqcnvcRSccl/ZDtU7b3ll8WAGDV\n0DXqiLi/ikIAAIOx9AEAyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAc\nQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0A\nyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJAcQQ0AyRHUAJBcoaC2faftr9h+zvZvl10U\nAOAVQ4Pa9g2SHpH005J2Srrf9s6yCwMA9BQ5ov4xSc9FxPMR8T+S2pLuKbcsAMCqmQJjbpb0b1c8\nPiXpx68eZHtR0mL/Ydf2V0asaVbSmRH3Hd0vP1z5lFeop+d6TVvP09avNJU9PzxOz9+/1oYiQV1I\nROyXtH/c57G9FBHNCZS0YdDz9W/a+pXoeZKKLH18XdItVzze0f8aAKACRYL6HyW90fYP2H6NpPsk\nHS63LADAqqFLHxHxsu1fk/RpSTdIejQini6xprGXTzYger7+TVu/Ej1PjCOijOcFAEwIVyYCQHIE\nNQAkV1tQD7ss3fZrbX+0v/3ztuerr3JyCvT7G7afsf2k7b+3veY5lRtF0Y8esP2ztsP2hj+Vq0jP\ntn+u/1o/bfvDVdc4aQV+tr/P9lHbT/R/vu+qo85Jsf2o7dO2n1pju23/Uf/78aTtN489aURUflPv\nTcmvSXqDpNdI+pKknVeN+VVJf9y/f5+kj9ZRa4X9tiR9Z//+uzZyv0V77o/bKumYpBOSmnXXXcHr\n/EZJT0j6nv7jm+quu4Ke90t6V//+Tkkv1F33mD3/hKQ3S3pqje13SfqkJEt6i6TPjztnXUfURS5L\nv0fSh/r3D0l6m21XWOMkDe03Io5GxMX+wxPqna++kRX96IHfk/QHkv67yuJKUqTnX5H0SESck6SI\nOF1xjZNWpOeQ9F39+98t6d8rrG/iIuKYpLPXGHKPpD+LnhOSttl+/Thz1hXUgy5Lv3mtMRHxsqQV\nSd9bSXWTV6TfK+1V71/kjWxoz/0/CW+JiCNVFlaiIq/zbZJus/052yds31lZdeUo0vN7JT1g+5Sk\nv5H0YDWl1Wa9v+9DTewSckyG7QckNSX9ZN21lMn2JkkfkPRLNZdStRn1lj8W1Pur6ZjtH4mIb9Za\nVbnul/TBiHjY9m5Jf2779oj437oL2yjqOqIucln6t8fYnlHvT6ZvVFLd5BW6DN/2T0l6j6S7I+JS\nRbWVZVjPWyXdLqlj+wX11vIOb/A3FIu8zqckHY6IlyLinyV9Vb3g3qiK9LxX0sckKSKOS9qs3gc2\nXa8m/rEbdQV1kcvSD0v6xf79eyV9Jvor9RvQ0H5tv0nSn6gX0ht93VIa0nNErETEbETMR8S8euvy\nd0fEUj3lTkSRn+u/Vu9oWrZn1VsKeb7KIiesSM//KultkmT7h9UL6v+qtMpqHZb0C/2zP94iaSUi\nXhzrGWt85/Qu9Y4mvibpPf2vvU+9X1ap92L+paTnJH1B0hvqfre35H7/TtJ/SjrZvx2uu+aye75q\nbEcb/KyPgq+z1VvyeUbSlyXdV3fNFfS8U9Ln1Dsj5KSkd9Rd85j9fkTSi5JeUu8vpL2S3inpnVe8\nxo/0vx9fnsTPNZeQA0ByXJkIAMkR1ACQHEENAMkR1ACQHEENAMkR1ACQHEENAMn9H1uDvBTu2BXL\nAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2KpMOhRwo1Xk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "cc56112a-6634-471e-a3e7-ec5d0c126785"
      },
      "source": [
        "train_x['delta_oper_start_time_oper_end_time'].max()"
      ],
      "execution_count": 309,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2966.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 309
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aDa2bivSgYFe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        },
        "outputId": "85a88536-88d9-45b8-dc36-7e2503090f49"
      },
      "source": [
        "\n",
        "plt.scatter(train_x[DUR_EN], train_x['delta_oper_start_time_oper_end_time'])\n",
        "plt.xlim(0, 3000)\n",
        "plt.ylim(0, 3000)"
      ],
      "execution_count": 310,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0, 3000)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 310
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAD8CAYAAACCRVh7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAbxUlEQVR4nO3df5Cd1X3f8feHZYE1pl4IsiotYoSx\nKgqDLZEdwIMnQ+0xAnUSZOra0DhWXSbKtDBjaqpGMrSAf1S4isHxDCGVAxOcEEBgsYhALCsYj2c6\ng2DllRACqywYGy4CEaMFHK3xsnz7xz1XXK322b0/nrv3x35eMzv73HOf+9xz5q7uR+c55zmPIgIz\nM7PJHNHsCpiZWetySJiZWSaHhJmZZXJImJlZJoeEmZllckiYmVmmaUNC0jGSHpe0U9JuSTek8lMk\nbZM0LOkeSUel8qPT4+H0/MKyY61N5XskLWtUo8zMLB+V9CTeBj4RER8FlgAXSjoX+CZwc0R8GNgP\nXJ72vxzYn8pvTvsh6XTgUuAM4ELgLyR15dkYMzPL17QhEUW/Tg+7008AnwDuS+V3ACvS9sXpMen5\nT0pSKr87It6OiJ8Dw8DZubTCzMwa4shKdkr/498OfBi4BXgOGImId9IuLwF9absPeBEgIt6R9Abw\nO6n8sbLDlr+m/L1WAasAjj322N897bTTqmySmdnsMHJgjFfe/A1j4+/S3XUE//JfHEPv+7rZvn37\nP0XEnDzeo6KQiIhxYImkXuB+oGHf3BGxAdgA0N/fH4ODg416KzOztjUwVGDtpl2cODZ+sKy7u4tr\nLzmTT5910i/yep+qZjdFxAjwKPAxoFdSKWROAgppuwAsAEjPfwD4VXn5JK8xM7MqrN+yh9GygAAY\nHRtn/ZY9ub5PJbOb5qQeBJJ6gE8Bz1AMi8+k3VYCD6Ttzekx6fkfRXEVwc3ApWn20ynAIuDxvBpi\nZjabvDwyWlV5rSo53TQPuCONSxwBbIyIv5f0NHC3pK8DQ8Btaf/bgL+RNAy8TnFGExGxW9JG4Gng\nHeCKdBrLzMyqNL+3h8IkgTC/t4cXcnwftfJS4R6TMDObXGlMovyUU093F+uKYxLbI6I/j/epaODa\nzMxay4qlxcmh67fs4eWRUeb39rB62eKD5XlxSJiZtakVS/tyD4WJvHaTmZllckiYmVkmh4SZmWVy\nSJiZWSaHhJmZZXJImJlZJoeEmZllckiYmVkmh4SZmWVySJiZWSaHhJmZZXJImJlZJi/wZ2ZWh4Gh\nQsNXYm0mh4SZWY0m3tOhMDLK2k27ADomKHy6ycysRjN1n+lmckiYmdVopu4z3UwOCTOzGs3v7amq\nvB05JMzMarR62WJ6ursOKevp7mL1ssVNqlH+PHBtZlajmbrPdDM5JMzM6jAT95luJp9uMjOzTA4J\nMzPL5JAwM7NMDgkzM8vkkDAzs0zThoSkBZIelfS0pN2SvpTKr5dUkLQj/Swve81aScOS9khaVlZ+\nYSoblrSmMU0yM7O8VDIF9h3g6oj4qaTjgO2Stqbnbo6IPyvfWdLpwKXAGcB84B8l/av09C3Ap4CX\ngCckbY6Ip/NoiJmZ5W/akIiIvcDetP2WpGeAqSYFXwzcHRFvAz+XNAycnZ4bjojnASTdnfZ1SJiZ\ntaiqxiQkLQSWAttS0ZWSnpR0u6TjU1kf8GLZy15KZVnlZmbWoioOCUnvB74PXBURbwK3AqcCSyj2\nNL6VR4UkrZI0KGnwtddey+OQZmZWo4pCQlI3xYC4MyI2AUTEqxExHhHvAt/lvVNKBWBB2ctPSmVZ\n5YeIiA0R0R8R/XPmzKm2PWZmlqNKZjcJuA14JiJuKiufV7bbp4Gn0vZm4FJJR0s6BVgEPA48ASyS\ndIqkoygObm/OpxlmZtYIlcxuOg/4I2CXpB2p7CvAZZKWAAG8APwJQETslrSR4oD0O8AVETEOIOlK\nYAvQBdweEbtzbIuZmeVMEdHsOmTq7++PwcHBZlfDzKytSNoeEf15HMtXXJuZWSaHhJmZZXJImJlZ\nJoeEmZllckiYmVkmh4SZmWVySJiZWSaHhJmZZXJImJlZJoeEmZllckiYmVkmh4SZmWVySJiZWSaH\nhJmZZXJImJlZpkpuOmRmVpOBoQLrt+zh5ZFR5vf2sHrZYlYs7Wt2tawKDgkza4iBoQJrN+1idGwc\ngMLIKGs37QJwULQRn24ys4ZYv2XPwYAoGR0bZ/2WPU2qkdXCIWFmDfHyyGhV5daaHBJm1hDze3uq\nKrfW5JAws4ZYvWwxPd1dh5T1dHexetniJtXIauGBazNriNLgtGc3tTeHhJk1zIqlfQ6FNufTTWZm\nlskhYWZmmRwSZmaWySFhZmaZHBJmZpZp2pCQtEDSo5KelrRb0pdS+QmStkp6Nv0+PpVL0nckDUt6\nUtJZZcdamfZ/VtLKxjXLzMzyUElP4h3g6og4HTgXuELS6cAa4JGIWAQ8kh4DXAQsSj+rgFuhGCrA\ndcA5wNnAdaVgMTOz1jRtSETE3oj4adp+C3gG6AMuBu5Iu90BrEjbFwPfi6LHgF5J84BlwNaIeD0i\n9gNbgQtzbY2ZmeWqqjEJSQuBpcA2YG5E7E1PvQLMTdt9wItlL3splWWVT3yPVZIGJQ2+9tpr1VTP\nzMxyVnFISHo/8H3gqoh4s/y5iAgg8qhQRGyIiP6I6J8zZ04ehzQzsxpVFBKSuikGxJ0RsSkVv5pO\nI5F+70vlBWBB2ctPSmVZ5WZm1qIqmd0k4DbgmYi4qeypzUBphtJK4IGy8i+kWU7nAm+k01JbgAsk\nHZ8GrC9IZWZm1qIqWeDvPOCPgF2SdqSyrwA3AhslXQ78Avhseu5hYDkwDBwAvggQEa9L+hrwRNrv\nqxHxei6tMDOzhlBxOKE19ff3x+DgYLOrYXaYgaFCWyyB3S71tHxJ2h4R/Xkcy0uFm1VpYKjA2k27\nDt6/uTAyytpNuwBa5gt4YKjADQ/uZv+BsYNlrVhPa31elsOsSuu37DkYECWjY+Os37KnSTU6VCnE\nygOipJXqae3BIWFWpZdHRqsqn2mThVi5VqmntQeHhFmV5vf2VFU+06YLgVapp7UHh4RZlVYvW0xP\nd9chZT3dXaxetrhJNTrUVCHQSvW09uCQMKvSiqV9rLvkTPp6exDQ19vDukvObJnB4MlCDKC3p7ul\n6mntwbObzGqwYmlfy37Zlurlqa+WB4eEWQdq5RCz9uLTTWZmlskhYWZmmRwSZmaWySFhZmaZHBJm\nZpbJIWFmZpkcEmZmlskhYWZmmXwxnVkTXDuwi7u2vch4BF0Sl52zgK+vOLPZ1TI7jEPCbAYNDBX4\nyqYnOTD27sGy8Qj+9rFfAjgorOX4dJPZDBkYKrD6vp2HBES5u7a9OMM1MpueQ8Jshtzw4G7GxrPv\nKT/ewvebt9nLIWE2Qya7nWi5LmmGamJWOYeEWYu47JwFza6C2WEcEmYzpLenO/O5z597sgetrSV5\ndpNZTqab1nr9H5zB6nt3Mvbue2MP3UeI9f/+o773g7Ush4RZDq4d2HVwGitMPq3Vd4yzdqRo4RkV\n/f39MTg42OxqmE3r1LUPTzo7qUviuXXLm1Ajm80kbY+I/jyO5TEJsxxkTV/1tFZrd9OGhKTbJe2T\n9FRZ2fWSCpJ2pJ/lZc+tlTQsaY+kZWXlF6ayYUlr8m+KWfNkTV/1tFZrd5X0JP4auHCS8psjYkn6\neRhA0unApcAZ6TV/IalLUhdwC3ARcDpwWdrXrCNkTV/1tFZrd9MOXEfETyQtrPB4FwN3R8TbwM8l\nDQNnp+eGI+J5AEl3p32frrrGZi2oNDjtRfus09Qzu+lKSV8ABoGrI2I/0Ac8VrbPS6kM4MUJ5edM\ndlBJq4BVACeffHId1TObWV9fcaZDwTpOrQPXtwKnAkuAvcC38qpQRGyIiP6I6J8zZ05ehzUzsxrU\n1JOIiFdL25K+C/x9elgAyk/CnpTKmKLcrKUMDBV8LYNZUlNISJoXEXvTw08DpZlPm4G/k3QTMB9Y\nBDwOCFgk6RSK4XAp8B/qqbhZ3gaGCqzd9CSjZUt5F0ZGWbtpF4CDwmalaUNC0l3A+cCJkl4CrgPO\nl7QECOAF4E8AImK3pI0UB6TfAa6IiPF0nCuBLUAXcHtE7M69NWY1GhgqHLZkRsno2Djrt+xxSNis\nVMnspssmKb5tiv2/AXxjkvKHgYerqp1Zgw0MFbjhwd3TLuP98sjoDNXIrLV47SabtSautzSV+b09\nDa6NWWtySNisM9l9pqciYPWyxY2tlFmLckjYrFJN76HkD8892eMRNms5JGxWGBgq8N/v28lvp7jH\n9ETHv6+b637/DAeEzWoOCet4A0MFvrxxB5NMXJqUbwRk9h6HhHWs0kVxhSpmJvV0H8G6Sz7igDBL\nHBLWcSqd1jqR7zNtdjiHhHWUWgam3Xswy+aQsI5RS0Ccd+oJ3PnHH2tQjczan0PCOsLAUKGqgJDg\nD8/x6SWz6TgkrK1VMzjd093FukvO9Gklsyo4JKwtDQwVuOqeHRXvL3BAmNWg1psOmTVNtQEBcPPn\nljggzGrgnoS1jXqmtjogzGrjkLCWNzBU4Jr7d/HPvx2v6nVeVsOsfg4Ja2kDQwWuvncn45WuqQF8\n26eWzHLjkLCWVct1D3OPO8oBYZYjD1xbS6olIBZ98Fi2XfOpBtXIbHZyT8JaxsBQges372ZktPKB\naa/YatZYDglruoGhAms3PclohXeKK/GaS2aN55CwphoYKrD63p2MVTEwDV6x1WymOCRsxpSW0Hh5\nZJTe93Xz69+MUWXnwb0HsxnmkLAZUTyltIvRseK1DtVeENclcdk5C9x7MJthDgmbEeu37DkYENXo\n7hLrP+OBabNm8RRYmxHV3EK05NijuhwQZk3mnoQ13Eeu+0FV+3s5DbPW4ZCwhqnlgjjPWjJrLdOe\nbpJ0u6R9kp4qKztB0lZJz6bfx6dySfqOpGFJT0o6q+w1K9P+z0pa2ZjmWCsYGCrwr//HP9R0xbQD\nwqy1VDIm8dfAhRPK1gCPRMQi4JH0GOAiYFH6WQXcCsVQAa4DzgHOBq4rBYt1jmsHdrFwzUNcdc+O\nqi6MO0LFHsTWL5/fuMqZWU2mPd0UET+RtHBC8cXA+Wn7DuDHwJ+m8u9FRACPSeqVNC/tuzUiXgeQ\ntJVi8NxVdwusJZx2zcP8Zry6C+LAK7aatbpaZzfNjYi9afsVYG7a7gNeLNvvpVSWVX4YSaskDUoa\nfO2112qsns2kD699yAFh1qHqHriOiJBU/TdE9vE2ABsA+vv7czuu5e9TN/2YZ/f9c9WvO+/UE7jz\njz/WgBqZWd5qDYlXJc2LiL3pdNK+VF4AFpTtd1IqK/De6alS+Y9rfG9rktKyGrVc8wDwvu4j+F9e\nUsOsrdR6umkzUJqhtBJ4oKz8C2mW07nAG+m01BbgAknHpwHrC1KZtYnSshq1BsS3P7eEp792kQPC\nrM1M25OQdBfFXsCJkl6iOEvpRmCjpMuBXwCfTbs/DCwHhoEDwBcBIuJ1SV8Dnkj7fbU0iG3t4ap7\ndtT8Wo89mLUvFScitab+/v4YHBxsdjVmvYVrHqrpdYs+eKyntZo1gaTtEdGfx7F8xbVlquWKaSje\nZ9q3ETXrDA4JO8zAUIGrN+6ghlmtXlbDrMN0dEiU3+Rmfm8Pq5ct9rnxaXzkuh/w5tvVL+kN8MKN\n/zbn2phZs3VsSEy8yU1hZJS1m3YBOCgmUeupJfDpJbNO1rEhMdlNbkbHxlm/Zc+sCYlKe1K1XhTn\ncDDrfB0bEi9nzOfPKq9Uu5zCqrQnVevpJV81bTY7dGxIzO/tmfTCr/m9PTUfs51OYU3Xk6o1HI4U\nDK/z2IPZbNGxty9dvWwxPd1dh5T1dHexetnimo851Rdvq8nqMRVGRlm45qGaAuLbn1vigDCbZTq2\nJ1H6n32ep4aqPYXVzFNTWT2pWnjswWz26tiQgGJQVPqlXMkXejWnsJp9amr1ssWHvH8tfGrJzDr2\ndFM1yhevC977Qh8YKhyyXzWnsJp9amrF0j7WXVL7RW2fP/dkB4SZdXZPolKVTpet5hRWo2ZXVaqe\ni+K8IJ+ZlTgkqO4LvdJTWI2YXVUJXxRnZnlySNCYL/TJxgTqnV1VbrIxlHqW8/Z1D2Y2GYcEjflC\nb8TsqpLJBsVrDQgv521mU3FI0Lgv9GpmV0001WyrycZQqnVMl/jZN5bXdQwz63y+6VALmthTABAQ\nQF8O1z/41JJZZ/NNhzrcZD2FUpTXGxBeztvMquHrJFpQI6bJzj3uKAeEmVXNPYkWlOeSGuDeg5nV\nzj2JOg0MFTjvxh9xypqHOO/GHx12lXYtJruyuxbf/twSB4SZ1cU9iTrktT7TZDOZ1l1yZs3TWj1z\nyczy4pCoQ6XLeUw1nTXPax6guObS11fUvmaTmVk5h0QdKlnOIysEbnhwN9f9/hm5XPNQ4lNLZpY3\nj0nU4QM93dOWZ4XA/gNjrL53Zy4D1J8/92QHhJk1hHsSdZCmL59qOuvYu/VfyOhwMLNGck+iDiMH\nxqYtb9Sqr+edeoIDwswarq6ehKQXgLeAceCdiOiXdAJwD7AQeAH4bETslyTgz4HlwAHgP0bET+t5\n/2YpDURn9QMCWPrVHxIBI6OTB0k9HA5mNlPy6En8m4hYUrZOyBrgkYhYBDySHgNcBCxKP6uAW3N4\n7xk3MFRg9X3TjyXsPzCWe0B47MHMZlojxiQuBs5P23cAPwb+NJV/L4orCj4mqVfSvIjY24A6NMTA\nUIEvb9xBDkMJVXM4mFkz1BsSAfxQUgD/JyI2AHPLvvhfAeam7T7gxbLXvpTK2iIkSlNZZzogHA5m\n1kz1hsTHI6Ig6YPAVkk/K38yIiIFSMUkraJ4OoqTTz65rspNdRFbtfK8nqFSDggza7a6QiIiCun3\nPkn3A2cDr5ZOI0maB+xLuxeABWUvPymVTTzmBmADFO8nUWvd8loyo6QRK7NmcTiYWauoOSQkHQsc\nERFvpe0LgK8Cm4GVwI3p9wPpJZuBKyXdDZwDvNHI8YhKlswo9TQKI6N0SYxH0JfR48h7ZdYsDggz\nayX19CTmAvcXZ7ZyJPB3EfEDSU8AGyVdDvwC+Gza/2GK01+HKU6B/WId7z2t6ZbMmNjTGE936Mvq\ncUx2H+w8ORzMrBXVHBIR8Tzw0UnKfwV8cpLyAK6o9f2qlfU//9LFbVONMYyOjXP1xp3813t2HDaW\nccODu9mfcRFdrRwQZtaqOnZZjsn+59/T3cXqZYuB6ccYsnoWK5b2HXKaqh4OBzNrdR27LMeKpX2s\nu+RM+np7ENDX28O6S8482COoZrmM0lhG+bHrCYgj5YAws/agiCZcGVah/v7+GBwcbMixJ45JVOq8\nU0/g/z73es3v63Aws0aTtL1sFYy6dGxPYjorlvbx7363j4yFXDPVGhDCAWFm7adjxyQq8ejPXstc\npC9PDgcza1ezticBjb9Azr0HM2t3s7on0cgL5BwOZtYJOj4kplq/qREXyDkczKyTdHRIZK3fdO/g\nL3ns+f0Hr4XIiwPCzDpNR4dE1vpN9UxhnYzDwcw6VUcPXHtBPjOz+nRsT2Jg6LBVyHPlcDCz2aAj\nexKl+1A3igPCzGaLjuxJ3PDgbsbG879MzuFgZrNNx/UkBoYKuS/lDQ4IM5ud2r4nce3ALu7a9iLj\nEXRJHH1ktasxTc3hYGazWVuHxLUDu/jbx3558PF4BAfG8jvN5IAws9murUPirm0vNuS4Dgczs6K2\nDglfMW1m1lhtPXCtHIcfHBBmZodr257EwFCBPDoSDgczs2xt25O44cHddR/DAWFmNrW27UnUcy2E\nw8HMrDJtFxIL1zxU1+sdEGZmlWurkKgnIBwOZmbVa4uQcO/BzKw5Wjokniq84d6DmVkTtfTspnpm\nuDogzMzqN+M9CUkXAn8OdAF/FRE35nl8h4OZWX5mtCchqQu4BbgIOB24TNLpeR3fAWFmlq+Z7kmc\nDQxHxPMAku4GLgaeruegDgczs8aY6ZDoA8qXbn0JOKd8B0mrgFUAdB3J3juuyjzYb18Z3g6gb+Zd\nzRlzIvBPza5EA7l97a2T29fJbQNYnNeBWm52U0RsADYASBp8e++z/U2uUsNIGowIt69NuX3tq5Pb\nBsX25XWsmZ7dVAAWlD0+KZWZmVkLmumQeAJYJOkUSUcBlwKbZ7gOZmZWoRk93RQR70i6EthCcQrs\n7REx1XKuG2amZk3j9rU3t699dXLbIMf2KXK+u5uZmXWOlr7i2szMmsshYWZmmVo2JCRdKGmPpGFJ\na5pdn1pJekHSLkk7StPSJJ0gaaukZ9Pv41O5JH0ntflJSWc1t/aHk3S7pH2Sniorq7o9klam/Z+V\ntLIZbZkoo23XSyqkz2+HpOVlz61NbdsjaVlZeUv+7UpaIOlRSU9L2i3pS6m8Uz6/rPa1/Wco6RhJ\nj0vamdp2Qyo/RdK2VM970oQgJB2dHg+n5xeWHWvSNmeKiJb7oTio/RzwIeAoYCdwerPrVWNbXgBO\nnFD2v4E1aXsN8M20vRz4B0DAucC2Ztd/kvb8HnAW8FSt7QFOAJ5Pv49P28e3aNuuB/7bJPuenv4u\njwZOSX+vXa38twvMA85K28cB/y+1o1M+v6z2tf1nmD6D96ftbmBb+kw2Apem8r8E/nPa/i/AX6bt\nS4F7pmrzVO/dqj2Jg8t3RMRvgdLyHZ3iYuCOtH0HsKKs/HtR9BjQK2leMyqYJSJ+Arw+obja9iwD\ntkbE6xGxH9gKXNj42k8to21ZLgbujoi3I+LnwDDFv9uW/duNiL0R8dO0/RbwDMVVEDrl88tqX5a2\n+QzTZ/Dr9LA7/QTwCeC+VD7xsyt9pvcBn5QkstucqVVDYrLlO6b6sFtZAD+UtF3FJUcA5kbE3rT9\nCjA3bbdru6ttT7u188p0uuX20qkY2rxt6fTDUor/I+24z29C+6ADPkNJXZJ2APsoBvNzwEhEvJN2\nKa/nwTak598Afoca2taqIdFJPh4RZ1Fc+fYKSb9X/mQU+4AdMw+509oD3AqcCiwB9gLfam516ifp\n/cD3gasi4s3y5zrh85ukfR3xGUbEeEQsobhSxdnAaTPxvq0aEh2zfEdEFNLvfcD9FD/cV0unkdLv\nfWn3dm13te1pm3ZGxKvpH+e7wHd5r2velm2T1E3xC/TOiNiUijvm85usfZ32GUbECPAo8DGKpwBL\nF0WX1/NgG9LzHwB+RQ1ta9WQ6IjlOyQdK+m40jZwAfAUxbaUZoSsBB5I25uBL6RZJecCb5SdBmhl\n1bZnC3CBpONT1/+CVNZyJowJfZri5wfFtl2aZpGcAiwCHqeF/3bTOenbgGci4qaypzri88tqXyd8\nhpLmSOpN2z3ApyiOuTwKfCbtNvGzK32mnwF+lHqJWW3O1swR+2lG85dTnJ3wHHBNs+tTYxs+RHEm\nwU5gd6kdFM8NPgI8C/wjcEK8N4PhltTmXUB/s9swSZvuothlH6N4PvPyWtoD/CeKg2bDwBeb3a4p\n2vY3qe5Ppn9g88r2vya1bQ9wUav/7QIfp3gq6UlgR/pZ3kGfX1b72v4zBD4CDKU2PAX8z1T+IYpf\n8sPAvcDRqfyY9Hg4Pf+h6dqc9eNlOczMLFOrnm4yM7MW4JAwM7NMDgkzM8vkkDAzs0wOCTMzy+SQ\nMDOzTA4JMzPL9P8Bg6+ewf4HT38AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ShRA-5yLpdIE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "7a48a945-9f00-4468-986d-f03778f41bc4"
      },
      "source": [
        "train_x.oper_time_deviation.hist(bins=100)"
      ],
      "execution_count": 307,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f0f29726cf8>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 307
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAQnElEQVR4nO3df6zddX3H8edrVLSrzhZxNw1tVswa\nFyYblhuo0Zg7yUohi2WJMRAyGsfsMjHRjGQrW7JuMhNdgtsgDtfNzpIgyPyxNoirXeXGbAlIUaQF\nxF6xhjaFTouwYjJX994f53PrWb2ll3N7zj0nfT6Sk/M97+/3fM/r25zy6vme772kqpAkndl+br4D\nSJLmn2UgSbIMJEmWgSQJy0CSBCyY7wC9Ovfcc2vFihU9PffFF19k0aJFpzdQH41S3lHKCubtp1HK\nCqOVdy5ZH3744e9X1et/ZkVVjeTt4osvrl7df//9PT93PoxS3lHKWmXefhqlrFWjlXcuWYHdNcN/\nUz1NJEmyDCRJloEkCctAkoRlIEnCMpAkYRlIkrAMJElYBpIkztAy2HPweVZs/CIrNn5xvqNI0lA4\nI8tAkvT/WQaSJMtAkmQZSJKwDCRJWAaSJCwDSRKWgSQJy0CShGUgSWIWZZBkeZL7kzye5LEkH2jz\nc5LsTLKv3S9p8yS5NclUkkeTrOra1/q2/b4k67vmFyfZ055za5L042AlSTObzSeDY8CNVXUBsBq4\nIckFwEZgV1WtBHa1xwBXACvbbQNwO3TKA9gEXApcAmyaLpC2zXu7nrd27ocmSZqtU5ZBVR2qqq+3\n5f8CngDOA9YBW9tmW4Gr2vI64I7qeABYnGQpcDmws6qOVNVzwE5gbVv3C1X1QFUVcEfXviRJA7Dg\n5WycZAXwZuBBYKyqDrVVzwBjbfk84Omupx1os5eaH5hhPtPrb6DzaYOxsTEmJydfTvzjxhbCjRce\nA+h5H4N09OjRkcgJo5UVzNtPo5QVRitvP7LOugySvBr4HPDBqnqh+7R+VVWSOq3JZlBVm4HNAOPj\n4zUxMdHTfm67cxu37Okc+v5re9vHIE1OTtLrsQ7aKGUF8/bTKGWF0crbj6yzupooySvoFMGdVfX5\nNn62neKh3R9u84PA8q6nL2uzl5ovm2EuSRqQ2VxNFOCTwBNV9bGuVduB6SuC1gPbuubXtauKVgPP\nt9NJO4A1SZa0L47XADvauheSrG6vdV3XviRJAzCb00RvBX4H2JPkkTb7E+AjwD1Jrge+B7y7rbsP\nuBKYAn4EvAegqo4kuRl4qG33oao60pbfB3wKWAh8qd0kSQNyyjKoqn8HTnbd/2UzbF/ADSfZ1xZg\nywzz3cCbTpVFktQf/gSyJMkykCRZBpIkLANJEpaBJAnLQJKEZSBJwjKQJGEZSJKwDCRJWAaSJCwD\nSRKWgSQJy0CShGUgScIykCRhGUiSsAwkSVgGkiQsA0kSloEkCctAkoRlIEnCMpAkYRlIkrAMJElY\nBpIkLANJEpaBJAnLQJKEZSBJwjKQJGEZSJKwDCRJWAaSJCwDSRKWgSQJy0CShGUgSWIWZZBkS5LD\nSfZ2zf48ycEkj7TblV3rbkoyleTJJJd3zde22VSSjV3z85M82OafSXL26TxASdKpzeaTwaeAtTPM\n/7qqLmq3+wCSXABcDfxqe87fJTkryVnAx4ErgAuAa9q2AB9t+/pl4Dng+rkckCTp5TtlGVTVV4Ej\ns9zfOuDuqvrvqvouMAVc0m5TVfVUVf0YuBtYlyTAO4DPtudvBa56mccgSZqjBXN47vuTXAfsBm6s\nqueA84AHurY50GYAT58wvxR4HfDDqjo2w/Y/I8kGYAPA2NgYk5OTPQUfWwg3Xth5yV73MUhHjx4d\niZwwWlnBvP00SllhtPL2I2uvZXA7cDNQ7f4W4HdPV6iTqarNwGaA8fHxmpiY6Gk/t925jVv2dA59\n/7W97WOQJicn6fVYB22UsoJ5+2mUssJo5e1H1p7KoKqenV5O8g/Ave3hQWB516bL2oyTzH8ALE6y\noH066N5ekjQgPV1ammRp18PfBqavNNoOXJ3klUnOB1YCXwMeAla2K4fOpvMl8/aqKuB+4F3t+euB\nbb1kkiT17pSfDJLcBUwA5yY5AGwCJpJcROc00X7g9wGq6rEk9wCPA8eAG6rqJ20/7wd2AGcBW6rq\nsfYSfwzcneQvgW8AnzxtRydJmpVTlkFVXTPD+KT/wa6qDwMfnmF+H3DfDPOn6FxtJEmaJ/4EsiTJ\nMpAkWQaSJCwDSRKWgSQJy0CShGUgScIykCRhGUiSsAwkSVgGkiQsA0kSloEkCctAkoRlIEnCMpAk\nYRlIkrAMJElYBpIkLANJEpaBJAnLQJKEZSBJwjKQJGEZSJKwDCRJWAaSJCwDSRKWgSQJy0CShGUg\nScIykCRhGUiSsAwkSVgGkiQsA0kSloEkCctAkoRlIEliFmWQZEuSw0n2ds3OSbIzyb52v6TNk+TW\nJFNJHk2yqus569v2+5Ks75pfnGRPe86tSXK6D1KS9NJm88ngU8DaE2YbgV1VtRLY1R4DXAGsbLcN\nwO3QKQ9gE3ApcAmwabpA2jbv7Xreia8lSeqzU5ZBVX0VOHLCeB2wtS1vBa7qmt9RHQ8Ai5MsBS4H\ndlbVkap6DtgJrG3rfqGqHqiqAu7o2pckaUAW9Pi8sao61JafAcba8nnA013bHWizl5ofmGE+oyQb\n6HziYGxsjMnJyd7CL4QbLzwG0PM+Buno0aMjkRNGKyuYt59GKSuMVt5+ZO21DI6rqkpSpyPMLF5r\nM7AZYHx8vCYmJnraz213buOWPZ1D339tb/sYpMnJSXo91kEbpaxg3n4apawwWnn7kbXXq4mebad4\naPeH2/wgsLxru2Vt9lLzZTPMJUkD1GsZbAemrwhaD2zrml/XripaDTzfTiftANYkWdK+OF4D7Gjr\nXkiyul1FdF3XviRJA3LK00RJ7gImgHOTHKBzVdBHgHuSXA98D3h32/w+4EpgCvgR8B6AqjqS5Gbg\nobbdh6pq+kvp99G5Ymkh8KV2kyQN0CnLoKquOcmqy2bYtoAbTrKfLcCWGea7gTedKockqX/8CWRJ\nkmUgSbIMJElYBpIkLANJEpaBJAnLQJKEZSBJwjKQJGEZSJKwDCRJWAaSJCwDSRKWgSQJy0CShGUg\nScIykCRhGUiSsAwkSVgGkiQsA0kSloEkCctAkoRlIEnCMpAkYRlIkrAMJElYBpIkLANJEpaBJAnL\nQJKEZSBJwjKQJGEZSJKwDCRJWAaSJCwDSRKWgSQJy0CSxBzLIMn+JHuSPJJkd5udk2Rnkn3tfkmb\nJ8mtSaaSPJpkVdd+1rft9yVZP7dDkiS9XKfjk8FvVNVFVTXeHm8EdlXVSmBXewxwBbCy3TYAt0On\nPIBNwKXAJcCm6QKRJA1GP04TrQO2tuWtwFVd8zuq4wFgcZKlwOXAzqo6UlXPATuBtX3IJUk6ibmW\nQQFfTvJwkg1tNlZVh9ryM8BYWz4PeLrruQfa7GRzSdKALJjj899WVQeT/CKwM8m3uldWVSWpOb7G\nca1wNgCMjY0xOTnZ037GFsKNFx4D6Hkfg3T06NGRyAmjlRXM20+jlBVGK28/ss6pDKrqYLs/nOQL\ndM75P5tkaVUdaqeBDrfNDwLLu56+rM0OAhMnzCdP8nqbgc0A4+PjNTExMdNmp3Tbndu4ZU/n0Pdf\n29s+BmlycpJej3XQRikrmLefRikrjFbefmTt+TRRkkVJXjO9DKwB9gLbgekrgtYD29ryduC6dlXR\nauD5djppB7AmyZL2xfGaNpMkDchcPhmMAV9IMr2fT1fVvyZ5CLgnyfXA94B3t+3vA64EpoAfAe8B\nqKojSW4GHmrbfaiqjswhlyTpZeq5DKrqKeDXZ5j/ALhshnkBN5xkX1uALb1mkSTNjT+BLEmyDCRJ\nloEkCctAkoRlIEnCMpAkYRlIkrAMJElYBpIkLANJEpaBJAnLQJKEZSBJwjKQJGEZSJKwDCRJWAaS\nJCwDSRKWgSQJy0CShGUgScIykCRhGUiSsAwkSVgGkiQsA0kSloEkCctAkoRlIEnCMpAkYRlIkrAM\nJElYBpIkLANJEpaBJAnLQJKEZSBJwjKQJGEZSJKwDCRJDFEZJFmb5MkkU0k2znceSTqTDEUZJDkL\n+DhwBXABcE2SC+Y3lSSdOYaiDIBLgKmqeqqqfgzcDayb50ySdMZYMN8BmvOAp7seHwAuPXGjJBuA\nDe3h0SRP9vh65wLfB8hHe9zDYB3POwJGKSuYt59GKSuMVt65ZP2lmYbDUgazUlWbgc1z3U+S3VU1\nfhoiDcQo5R2lrGDefhqlrDBaefuRdVhOEx0Elnc9XtZmkqQBGJYyeAhYmeT8JGcDVwPb5zmTJJ0x\nhuI0UVUdS/J+YAdwFrClqh7r40vO+VTTgI1S3lHKCubtp1HKCqOV97RnTVWd7n1KkkbMsJwmkiTN\nI8tAknTmlcEw/tqLJFuSHE6yt2t2TpKdSfa1+yVtniS3tvyPJlk14KzLk9yf5PEkjyX5wLDmTfKq\nJF9L8s2W9S/a/PwkD7ZMn2kXLZDkle3xVFu/YlBZT8h9VpJvJLl3mPMm2Z9kT5JHkuxus6F7H3Tl\nXZzks0m+leSJJG8Z1rxJ3tj+XKdvLyT5YF/zVtUZc6Pz5fR3gDcAZwPfBC4YglxvB1YBe7tmfwVs\nbMsbgY+25SuBLwEBVgMPDjjrUmBVW34N8G06v0Jk6PK213x1W34F8GDLcA9wdZt/AviDtvw+4BNt\n+WrgM/P0fvhD4NPAve3xUOYF9gPnnjAbuvdBV7atwO+15bOBxcOctyv3WcAzdH5YrG955+Xg5vEP\n9S3Ajq7HNwE3zXeulmXFCWXwJLC0LS8FnmzLfw9cM9N285R7G/Cbw54X+Hng63R+sv37wIIT3xN0\nrmZ7S1te0LbLgHMuA3YB7wDubX+5hzLvScpgKN8HwGuB75745zOseU/IuAb4j37nPdNOE830ay/O\nm6cspzJWVYfa8jPAWFsemmNopyXeTOdf3EOZt51yeQQ4DOyk88nwh1V1bIY8x7O29c8DrxtU1uZv\ngD8C/rc9fh3Dm7eALyd5OJ1fFQND+j4Azgf+E/indgruH5MsYnjzdrsauKst9y3vmVYGI6k6VT9U\n1wAneTXwOeCDVfVC97phyltVP6mqi+j8i/sS4FfmOdJJJfkt4HBVPTzfWWbpbVW1is5vG74hydu7\nVw7T+4DOJ6dVwO1V9WbgRTqnWY4bsrwAtO+H3gn884nrTnfeM60MRunXXjybZClAuz/c5vN+DEle\nQacI7qyqz7fx0OYFqKofAvfTOc2yOMn0D1x25zmeta1/LfCDAcZ8K/DOJPvp/ObedwB/O6x5q+pg\nuz8MfIFO2Q7r++AAcKCqHmyPP0unHIY177QrgK9X1bPtcd/ynmllMEq/9mI7sL4tr6dzbn56fl27\nemA18HzXx8a+SxLgk8ATVfWxYc6b5PVJFrflhXS+23iCTim86yRZp4/hXcBX2r++BqKqbqqqZVW1\ngs578ytVde0w5k2yKMlrppfpnNfeyxC+DwCq6hng6SRvbKPLgMeHNW+Xa/jpKaLpXP3JOx9fiMzn\njc637t+mc+74T+c7T8t0F3AI+B86/4K5ns65313APuDfgHPatqHzPwL6DrAHGB9w1rfR+Wj6KPBI\nu105jHmBXwO+0bLuBf6szd8AfA2YovPx+5Vt/qr2eKqtf8M8vicm+OnVREOXt2X6Zrs9Nv13aRjf\nB12ZLwJ2t/fDvwBLhjzvIjqf9F7bNetbXn8dhSTpjDtNJEmagWUgSbIMJEmWgSQJy0CShGUgScIy\nkCQB/wfYX+GEQE9PHAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zjeGernyHK9c",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "adea922b-0b6b-4555-f109-24534a52a62e"
      },
      "source": [
        "ff =  (train_x.oper_end_time_abs > 0)\n",
        "\n",
        "xx1 = train_x[ff & (train_y==1)]\n",
        "xx2 = train_x[ff& (train_y==0)]\n",
        "\n",
        "c, b, _ = plt.hist(xx1.oper_end_time_abs, bins=25, alpha=0.5, density=True)\n",
        "c, b, _ = plt.hist(xx2.oper_end_time_abs, bins=b, alpha=0.5, density=True);\n",
        "plt.legend(['fraud', 'norm'])"
      ],
      "execution_count": 513,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f0f23aba1d0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 513
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAD4CAYAAAApWAtMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAaSklEQVR4nO3dfZBV1Znv8e8PEFpNBrGlLBWUZsQr\nEFKCrSHlNUaNipMETEWFzOjAlWhptOZG69ZVKrdMxdFKMM5FTYxI1ESpxNaoNXYsI4UBJ2++0ERN\nBhRpkGhTmRumeZnci4ANz/1jL5iT4znd5/Tu9/P7VHX1Pmuv/ey1N5vz9Np7nXUUEZiZmeUxrL8b\nYGZmg5+TiZmZ5eZkYmZmuTmZmJlZbk4mZmaW24j+bkB/OOaYY2LChAn93Qwzs0Fl7dq1/x4RY0ut\nq8lkMmHCBFpaWvq7GWZmg4qkP5Rb59tcZmaWm5OJmZnl5mRiZma51eQzk1I++OAD2tra2LNnT383\npU/V1dUxbtw4DjvssP5uipkNYk4mSVtbGx/96EeZMGECkvq7OX0iImhvb6etrY2Ghob+bo6ZDWK+\nzZXs2bOH+vr6mkkkAJKor6+vud6YmfU8J5MCtZRIDqrFYzaznudkYmZmufmZSRlLVr7do/FuvOCU\niurde++93H///cyYMYMf/ehHPbb/F198kbvuuotnn322x2KamR3kZDLAfO973+OFF15g3Lhxh8o6\nOjoYMcL/VH2pO39MVPoHg9lQ5NtcA8i1117L5s2bufjiixk9ejRXXnklZ511FldeeSVbtmzh7LPP\nZsaMGcyYMYPf/OY3QNbj+NznPncoxg033MAPf/hDAJ5//nlOPfVUZsyYwdNPP90fh2RmNcJ/7g4g\nS5cu5fnnn2f16tV897vf5ac//Sm/+tWvOPzww9m9ezcrV66krq6OjRs38qUvfanT+cX27NnD1Vdf\nzapVqzj55JOZO3duHx6JmdUa90wGsNmzZ3P44YcD2Ycqr776aqZNm8Zll13G+vXrO932rbfeoqGh\ngUmTJiGJK664oi+abGY1yj2TAezII488tLxkyRKOPfZY3njjDQ4cOEBdXR0AI0aM4MCBA4fq+TMj\nZtYf3DMZJHbt2sVxxx3HsGHDWL58Ofv37wfgpJNOYv369ezdu5edO3fy85//HIBTTz2VLVu2sGnT\nJgAee+yxfmu7mQ197pmUMdBG5nzlK1/hi1/8Io8++iizZs061GsZP348l19+OR/72MdoaGhg+vTp\nQDbn1rJly/jsZz/LEUccwdlnn82f//zn/jwEMxvCFBH93YY+19jYGMUPr998800mT57cTy3qX7V8\n7OV4aLDZh0laGxGNpdb5NpeZmeXm21xmNnSs/mZ19c9d1DvtqEFOJmY9xW9kVsOcTMzMKlXtHwxQ\nM380+JmJmZnl5mRiZma5VXSbS9Is4B5gOPBgRHyraP0o4FHgdKAdmBsRW9K6RcBCYD/wDxGxorOY\nkhqAJqAeWAtcGRH7JN0EfBnoALYBV0XEH9I2+4Hfp+a8GxGzqz8VRbrTne1MjXR1zaw2dZlMJA0H\n7gMuANqANZKaI6JwcqiFwI6IOFnSPGAxMFfSFGAeMBU4HnhB0sHB+OViLgaWRESTpKUp9v3Aa0Bj\nROyWdB1wJ3Bw9sL3I+K0HOdh0PM09Z3r6e+nMbO/VMltrjOB1ojYHBH7yHoNc4rqzAEeSctPAucr\n+z7YOUBTROyNiHeA1hSvZMy0zXkpBinmJQARsToidqfyl4H//MKPIWLLli1MnjyZq6++mqlTp3Lh\nhRfy/vvv8/rrrzNz5kw+/vGP84UvfIEdO3YA8OlPf5qvfvWrNDY2cs8997BgwQKuu+46Zs6cycSJ\nE3nxxRe56qqrmDx5MgsWLOjfgzOzIa2SP2VPAN4reN0GfKJcnYjokLSL7DbVCWRv/IXbnpCWS8Ws\nB3ZGREeJ+oUWAj8reF0nqYXsFti3IuKfizeQdA1wDcCJJ55Y8kAHgo0bN/LYY4/x/e9/n8svv5yn\nnnqKO++8k+985zucc8453HrrrXzjG9/g7rvvBmDfvn2HpqJfsGABO3bs4KWXXqK5uZnZs2fz61//\nmgcffJAzzjiD119/ndNOq+kO3MDiocQ2hAy6B/CSrgAagW8XFJ+UPuL/t8Ddkv66eLuIWBYRjRHR\nOHbs2D5qbfUaGhoOveGffvrpbNq0iZ07d3LOOecAMH/+fH7xi18cql/8PSWf//znkcS0adM49thj\nmTZtGsOGDWPq1Kls2bKlz47DzGpLJclkKzC+4PW4VFayjqQRwGiyB/Hlti1X3g4clWJ8aF+SPgN8\nDZgdEXsPlkfE1vR7M/AiML2C4xqQRo0adWh5+PDh7Ny5s9P6hdPUF24/bNiwv4g1bNgwOjo6MDPr\nDZUkkzXAJEkNkkaSPVBvLqrTDMxPy5cCqyKbQbIZmCdpVBqlNQl4tVzMtM3qFIMU8xkASdOBB8gS\nyZ8O7ljSmDSaDEnHAGcBnX9z1CAyevRoxowZwy9/+UsAli9ffqiXYmY2UHT5zCQ9A7kBWEE2jPfh\niFgn6TagJSKagYeA5ZJage1kyYFU7wmyN/cO4PqI2A9QKmba5c1Ak6TbyUZwPZTKvw18BPhJ9pz+\n0BDgycADkg6QJcdvFY00654BdH/6kUce4dprr2X37t1MnDiRH/zgB/3dJDOzv+Ap6JNanoa9Fo69\nL4YG3zjiqd7dwQD6A2fA6unPh/WEIfTv5inozcysVzmZmJlZbk4mBWrxll8tHrOZ9Twnk6Suro72\n9vaaenONCNrb26mrq+vvppjZIOfJnJJx48bR1tbGtm3b+rspfaquro5x44bczDRm1secTJLDDjuM\nhoaG/m6Gmdmg5NtcZmaWm5OJmZnl5mRiZma5OZmYmVluTiZmZpabk4mZmeXmZGJmZrk5mZiZWW5O\nJmZmlpuTiZmZ5eZkYmZmuTmZmJlZbk4mZmaWm5OJmZnl5mRiZma5OZmYmVlu/nIss0Fiycq3q6p/\n4wWn9FJLzD7MPRMzM8vNycTMzHLzbS6zoWr1N6urf+6i3mmH1QT3TMzMLDcnEzMzy83JxMzMcqvo\nmYmkWcA9wHDgwYj4VtH6UcCjwOlAOzA3IrakdYuAhcB+4B8iYkVnMSU1AE1APbAWuDIi9km6Cfgy\n0AFsA66KiD+kbeYD/ys15/aIeKT6U2GWz0ub26uq/8mJ9b3UErO+12UykTQcuA+4AGgD1khqjoj1\nBdUWAjsi4mRJ84DFwFxJU4B5wFTgeOAFSQcHv5eLuRhYEhFNkpam2PcDrwGNEbFb0nXAnWkfRwNf\nBxqBANamWDvynBgz61y1n3sBf/ZlKKvkNteZQGtEbI6IfWS9hjlFdeYAB3sDTwLnS1Iqb4qIvRHx\nDtCa4pWMmbY5L8UgxbwEICJWR8TuVP4yMC4tXwSsjIjtKYGsBGZVfgrMzCyvSpLJCcB7Ba/bUlnJ\nOhHRAewiu01Vbtty5fXAzhSj3L4g6638rIr2IekaSS2SWrZt21byQM3MrHsG3edMJF1BdkvrnGq2\ni4hlwDKAxsbG6IWmmVWl2mcsM7PLt3JVPpPpzm0rs4Mq6ZlsBcYXvB6XykrWkTQCGE32IL7ctuXK\n24GjUowP7UvSZ4CvAbMjYm8V7TMzs15USTJZA0yS1CBpJNkD9eaiOs3A/LR8KbAqIiKVz5M0Ko3S\nmgS8Wi5m2mZ1ikGK+QyApOnAA2SJ5E8F+14BXChpjKQxwIWpzMzM+kiXt7kiokPSDWRv0MOBhyNi\nnaTbgJaIaAYeApZLagW2kyUHUr0ngPVkQ3qvj4j9AKVipl3eDDRJup1sBNdDqfzbwEeAn2TP6Xk3\nImZHxHZJ/0iWoABui4jtOc6JmZlVqaJnJhHxHPBcUdmtBct7gMvKbHsHcEclMVP5ZrLRXsXln+mk\nfQ8DD5c/AjMz603+BLyZmeU26EZzmVllqh0txom90w6rDe6ZmJlZbk4mZmaWm5OJmZnl5mcmZtYt\nM9+t8hP5ANzV4+2wgcE9EzMzy83JxMzMcnMyMTOz3PzMxMwGrtXf7O8WWIWcTMwM6O4DdbOMb3OZ\nmVlu7pmYmfWmam/Vnbuod9rRy9wzMTOz3JxMzMwsNycTMzPLzc9MzGzAqnYa/U9OrO+lllhX3DMx\nM7PcnEzMzCw3JxMzM8vNycTMzHJzMjEzs9w8msvM+sySlW9XVX9mL7XDep57JmZmlpuTiZmZ5eZk\nYmZmuTmZmJlZbn4Ab2Z9xl/ANXS5Z2JmZrlVlEwkzZK0QVKrpFtKrB8l6fG0/hVJEwrWLUrlGyRd\n1FVMSQ0pRmuKOTKVf0rSbyV1SLq0aP/7Jb2efpqrPw1mZpZHl7e5JA0H7gMuANqANZKaI2J9QbWF\nwI6IOFnSPGAxMFfSFGAeMBU4HnhB0ilpm3IxFwNLIqJJ0tIU+37gXWAB8D9KNPP9iDitymM3Mxt4\nBuk3M1bSMzkTaI2IzRGxD2gC5hTVmQM8kpafBM6XpFTeFBF7I+IdoDXFKxkzbXNeikGKeQlARGyJ\niN8BB7p5rGZm1ksqSSYnAO8VvG5LZSXrREQHsAuo72TbcuX1wM4Uo9y+SqmT1CLpZUmXlKog6ZpU\np2Xbtm0VhDQzs0oNlQfwJ0VEI/C3wN2S/rq4QkQsi4jGiGgcO3Zs37fQzGwIqySZbAXGF7wel8pK\n1pE0AhgNtHeybbnyduCoFKPcvj4kIram35uBF4HpXR+WmZn1lEqSyRpgUhplNZLsgXrxiKlmYH5a\nvhRYFRGRyuel0V4NwCTg1XIx0zarUwxSzGc6a5ykMZJGpeVjgLOA9Z1tY2ZmPavLZJKeX9wArADe\nBJ6IiHWSbpM0O1V7CKiX1ArcBNyStl0HPEH25v48cH1E7C8XM8W6GbgpxapPsZF0hqQ24DLgAUkH\n608GWiS9QZaIvlU00szMzHpZRZ+Aj4jngOeKym4tWN5D9iZfats7gDsqiZnKN5ON9iouX0N226u4\n/DfAtC4PwszMes1QeQBvZmb9yMnEzMxyczIxM7PcnEzMzCw3T0Fvg1K13yVuZr3LPRMzM8vNPROz\nEvwlTmbVcc/EzMxyczIxM7PcnEzMzCw3JxMzM8vNycTMzHJzMjEzs9ycTMzMLDcnEzMzy83JxMzM\ncnMyMTOz3JxMzMwsNycTMzPLzcnEzMxy86zBZjZkvLS5var6n5xY30stqT3umZiZWW5OJmZmlpuT\niZmZ5eZkYmZmuTmZmJlZbk4mZmaWm5OJmZnl5mRiZma5VZRMJM2StEFSq6RbSqwfJenxtP4VSRMK\n1i1K5RskXdRVTEkNKUZrijkylX9K0m8ldUi6tGj/8yVtTD/zqz8NZmaWR5fJRNJw4D7gYmAK8CVJ\nU4qqLQR2RMTJwBJgcdp2CjAPmArMAr4naXgXMRcDS1KsHSk2wLvAAuDHRe07Gvg68AngTODrksZU\negLMzCy/SnomZwKtEbE5IvYBTcCcojpzgEfS8pPA+ZKUypsiYm9EvAO0pnglY6ZtzksxSDEvAYiI\nLRHxO+BA0b4vAlZGxPaI2AGsJEtcZmbWRypJJicA7xW8bktlJetERAewC6jvZNty5fXAzhSj3L66\n0z4kXSOpRVLLtm3bughpZmbVqJkH8BGxLCIaI6Jx7Nix/d0cM7MhpZJkshUYX/B6XCorWUfSCGA0\n0N7JtuXK24GjUoxy++pO+8zMrBdVkkzWAJPSKKuRZA/Um4vqNAMHR1FdCqyKiEjl89JorwZgEvBq\nuZhpm9UpBinmM120bwVwoaQx6cH7hanMzMz6SJfJJD2/uIHsDfpN4ImIWCfpNkmzU7WHgHpJrcBN\nwC1p23XAE8B64Hng+ojYXy5minUzcFOKVZ9iI+kMSW3AZcADktalfWwH/pEsQa0BbktlZmbWR5R1\nBmpLY2NjtLS09HczLIclK9/u1fgz313Wq/FtYBgSX4517qI+25WktRHRWGqdv2nRzGqWv5mx59TM\naC4zM+s9TiZmZpabk4mZmeXmZGJmZrk5mZiZWW5OJmZmlpuTiZmZ5eZkYmZmuTmZmJlZbk4mZmaW\nm5OJmZnl5mRiZma5OZmYmVluTiZmZpabk4mZmeXm7zMxM6tQtd9/0h2D9TtT3DMxM7PcnEzMzCw3\n3+ayAaG3v9PdzHqXeyZmZpabk4mZmeXmZGJmZrk5mZiZWW5OJmZmlpuTiZmZ5eZkYmZmuTmZmJlZ\nbk4mZmaWW0XJRNIsSRsktUq6pcT6UZIeT+tfkTShYN2iVL5B0kVdxZTUkGK0ppgjO9uHpAmS3pf0\nevpZ2t2TYWZm3dPldCqShgP3ARcAbcAaSc0Rsb6g2kJgR0ScLGkesBiYK2kKMA+YChwPvCDplLRN\nuZiLgSUR0ZQSw0Lg/nL7SLE2RcRpOc6DmdngtPqb1dU/d1GvNKOSnsmZQGtEbI6IfUATMKeozhzg\nkbT8JHC+JKXypojYGxHvAK0pXsmYaZvzUgxSzEu62IeZmfWzSpLJCcB7Ba/bUlnJOhHRAewC6jvZ\ntlx5PbAzxSjeV7l9ADRIek3Sv0g6u9RBSLpGUouklm3btlVw2GZmVqmh8AD+j8CJETEduAn4saS/\nKq4UEcsiojEiGseOHdvnjTQzG8oqmYJ+KzC+4PW4VFaqTpukEcBooL2LbUuVtwNHSRqReh+F9Uvu\nIyIC2AsQEWslbQJOAVoqOLZuqXa69BsvOKXrSmZmg1glPZM1wKQ0ymok2QP15qI6zcD8tHwpsCq9\nyTcD89JIrAZgEvBquZhpm9UpBinmM53tQ9LYNEgASRPTPjZXfgrMzCyvLnsmEdEh6QZgBTAceDgi\n1km6DWiJiGbgIWC5pFZgO1lyINV7AlgPdADXR8R+gFIx0y5vBpok3Q68lmJTbh/Ap4DbJH0AHACu\njYjt3T8lZmb9p9rvmR8o3xlf0TctRsRzwHNFZbcWLO8BLiuz7R3AHZXETOWbyUZ7FZeX3EdEPAU8\n1eVBmJlZrxkKD+DNzKyfOZmYmVluTiZmZpZbRc9MLB8PJTazoc49EzMzy83JxMzMcvNtLjOzQazq\nz6Wc2zvtcM/EzMxyc8/EekW1gw7MbHBzz8TMzHJzMjEzs9ycTMzMLDcnEzMzy83JxMzMcnMyMTOz\n3Dw0eAAaiHN5eaivmXXGyaQGOTGYWU/zbS4zM8vNycTMzHLzba4hwLetzKy/uWdiZma5OZmYmVlu\nTiZmZpabk4mZmeXmZGJmZrk5mZiZWW5OJmZmlpuTiZmZ5eZkYmZmuVWUTCTNkrRBUqukW0qsHyXp\n8bT+FUkTCtYtSuUbJF3UVUxJDSlGa4o5srv7MDOzvtHldCqShgP3ARcAbcAaSc0Rsb6g2kJgR0Sc\nLGkesBiYK2kKMA+YChwPvCDp4Hzp5WIuBpZERJOkpSn2/dXuIyL25zkxnZn57rKq6r984jW91BIz\ns4Ghkp7JmUBrRGyOiH1AEzCnqM4c4JG0/CRwviSl8qaI2BsR7wCtKV7JmGmb81IMUsxLurkPMzPr\nI5VM9HgC8F7B6zbgE+XqRESHpF1AfSp/uWjbE9JyqZj1wM6I6ChRvzv7OETSNcDBLsL/lbSh/CF3\n6Rjg3yuv/k85djWoVHleaorPTWk+L6X13nn5cq73o5PKraiZWYMjYhlQ3f2pMiS1RERjT8QaSnxe\nyvO5Kc3npbTBeF4quc21FRhf8HpcKitZR9IIYDTQ3sm25crbgaNSjOJ9VbsPMzPrI5UkkzXApDTK\naiTZw+7mojrNwPy0fCmwKiIilc9LI7EagEnAq+Vipm1WpxikmM90cx9mZtZHurzNlZ5P3ACsAIYD\nD0fEOkm3AS0R0Qw8BCyX1ApsJ0sOpHpPAOuBDuD6g6OsSsVMu7wZaJJ0O/Baik139tGLeuR22RDk\n81Kez01pPi+lDbrzouyPezMzs+7zJ+DNzCw3JxMzM8vNyaQKXU0rMxRIGi9ptaT1ktZJ+u+p/GhJ\nKyVtTL/HpHJJujedk99JmlEQa36qv1HS/ILy0yX9Pm1zb/rw6aAgabik1yQ9m1732PQ/g/n6knSU\npCclvSXpTUmf9DUDkm5M/4/+VdJjkuqG7DUTEf6p4IdsoMAmYCIwEngDmNLf7eqF4zwOmJGWPwq8\nDUwB7gRuSeW3AIvT8t8APwMEzAReSeVHA5vT7zFpeUxa92qqq7Ttxf193FWcn5uAHwPPptdPAPPS\n8lLgurT8FWBpWp4HPJ6Wp6RrZxTQkK6p4YP9+iKbneLLaXkkcFStXzNkH55+Bzi84FpZMFSvGfdM\nKlfJtDKDXkT8MSJ+m5b/DLxJ9p+icDqb4mluHo3My2SfEzoOuAhYGRHbI2IHsBKYldb9VUS8HNn/\nlEcLYg1oksYBnwUeTK97cvqfQXt9SRoNfIo08jIi9kXETnzNQDZi9nBln407AvgjQ/SacTKpXKlp\nZT40bctQkrrZ04FXgGMj4o9p1b8Bx6blcuels/K2EuWDwd3A/wQOpNcVT/8DFE7/U835GgwagG3A\nD9ItwAclHUmNXzMRsRW4C3iXLInsAtYyRK8ZJxMrSdJHgKeAr0bEfxSuS38d1tSYckmfA/4UEWv7\nuy0D0AhgBnB/REwH/h/Zba1DavSaGUPWU2ggm9H8SGBWvzaqFzmZVK5mpm2RdBhZIvlRRDydiv9P\nut1A+v2nVF7tlDlb03Jx+UB3FjBb0hay2wnnAffQc9P/DObrqw1oi4hX0usnyZJLrV8znwHeiYht\nEfEB8DTZdTQkrxknk8pVMq3MoJfu0T4EvBkR/7tgVeF0NsXT3Px9GqEzE9iVbm2sAC6UNCb9hXYh\nsCKt+w9JM9O+/r4g1oAVEYsiYlxETCD7t18VEX9Hz03/M2ivr4j4N+A9Sf8lFZ1PNiNFTV8zZLe3\nZko6IrX74HkZmtdMfz35H4w/ZKNQ3iYbQfG1/m5PLx3jfyW7HfE74PX08zdk925/DmwEXgCOTvVF\n9kVnm4DfA40Fsa4ie1jYCvy3gvJG4F/TNt8lzcQwWH6AT/Ofo7kmkv3HbgV+AoxK5XXpdWtaP7Fg\n+6+lY99AwaikwXx9AacBLem6+Wey0Vg1f80A3wDeSm1fTjYia0heM55OxczMcvNtLjMzy83JxMzM\ncnMyMTOz3JxMzMwsNycTMzPLzcnEzMxyczIxM7Pc/j9f1xBO/qShdwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xeaBnlmIGfCI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "6eef069f-0a01-4d2c-8cc3-21cf55fe534b"
      },
      "source": [
        "xx1 = train_x[(train_x.delta_queue_time_oper_start_time< 50) & (train_y==1)]\n",
        "xx2 = train_x[(train_x.delta_queue_time_oper_start_time< 50) & (train_y==0)]\n",
        "\n",
        "c, b, _ = plt.hist(xx1.delta_queue_time_oper_start_time, bins=25, alpha=0.5, density=True)\n",
        "c, b, _ = plt.hist(xx2.delta_queue_time_oper_start_time, bins=b, alpha=0.5, density=True);\n",
        "plt.legend(['fraud', 'norm'])"
      ],
      "execution_count": 484,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f0f2379ecc0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 484
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAARIklEQVR4nO3df5BV5X3H8fd3AV1iHCRIqcMSWZWU\nkJAiUo2jVuMkLVQDmfEnU4mZSXWYyIw1dlrTdmy0k0lNnNA6mh9MQhTGaG00lViqJUpGGUnKGn8Q\nQMKP2cR10kCIGDMM4sq3f9wLudks7F327q9n368ZZu8557n3fp/l8tmH55zzbGQmkqThr2mwC5Ak\nNYaBLkmFMNAlqRAGuiQVwkCXpEKMHqw3Pvnkk3Pq1KmD9faSNCw999xzv8zMid0dG7RAnzp1Km1t\nbYP19pI0LEXET490zCkXSSqEgS5JhTDQJakQgzaHLkn1eOutt+jo6GD//v2DXcqAam5upqWlhTFj\nxtT9HANd0pDW0dHBiSeeyNSpU4mIwS5nQGQme/bsoaOjg9bW1rqf55SLpCFt//79TJgwYcSEOUBE\nMGHChF7/r8RAlzTkjaQwP+RY+mygS1IhnEOXNKwsXfOThr7eTR95T13t7rrrLr7yla8we/Zs7r//\n/oa9//e//33uvPNOHnvssT6/1vAM9LWf7137D32mf+qQNGJ8+ctf5nvf+x4tLS2H93V2djJ69NCJ\nUadcJKkHixcvZufOncybN49x48axaNEizjvvPBYtWkR7ezsXXHABs2fPZvbs2Tz77LNAZeR96aWX\nHn6NJUuWcO+99wLw+OOPM336dGbPns0jjzzSsDqHzo8WSRqivvrVr/L444+zdu1a7r77br773e+y\nbt06xo4dy759+1izZg3Nzc1s27aNhQsXHnWdqv3793Pdddfx1FNPccYZZ3DVVVc1rE5H6JLUS/Pn\nz2fs2LFA5can6667jpkzZ3LFFVewefPmoz735ZdfprW1lWnTphERXHPNNQ2ryxG6JPXSCSeccPjx\n0qVLmTRpEi+++CIHDx6kubkZgNGjR3Pw4MHD7QbiTldH6JLUB6+//jqnnHIKTU1NrFy5krfffhuA\nU089lc2bN/Pmm2+yd+9ennzySQCmT59Oe3s7O3bsAOCBBx5oWC2O0CUNK/VeZjhQPvWpT3HZZZex\nYsUK5s6de3j0PmXKFK688kre//7309rayplnnglU1mhZtmwZl1xyCe94xzu44IILeOONNxpSS2Rm\nQ16ot+bMmZPH/AsuvGxRGjG2bNnCe9/73sEuY1B01/eIeC4z53TX3ikXSSqEgS5JhTDQJakQBrok\nFcJAl6RCGOiSVAivQ5c0vPT2suWeFHRZsyN0SepHnZ2dA/ZejtAlqQft7e3MmzeP888/n2effZbJ\nkyfz6KOPsnXrVhYvXsy+ffs4/fTTWb58OePHj+eiiy5i1qxZrFu3joULF7Jx40bGjh3L888/z65d\nu1i+fDkrVqxg/fr1nHPOOYeX1e2rukboETE3IrZGxPaIuOUo7S6LiIyIbu9ikqThatu2bdxwww1s\n2rSJk046iYcffpiPf/zj3HHHHbz00kvMnDmT22677XD7AwcO0NbWxs033wzAa6+9xvr161m6dCnz\n58/npptuYtOmTWzcuJEXXnihITX2GOgRMQq4B5gHzAAWRsSMbtqdCNwI/LAhlUnSENLa2sqsWbMA\nOOuss9ixYwd79+7lwgsvBODaa6/l6aefPty+6zrnH/3oR4kIZs6cyaRJk5g5cyZNTU28733vo729\nvSE11jNCPxvYnpk7M/MA8CCwoJt2/wzcAfT/GpGSNMCOP/74w49HjRrF3r17j9q+dond2uc3NTX9\nzms1NTU1bJ69nkCfDLxSs91R3XdYRMwGpmTmfx3thSLi+ohoi4i23bt397pYSRoqxo0bx/jx43nm\nmWcAWLly5eHR+mDp80nRiGgCvgR8oqe2mbkMWAaV1Rb7+t6SRqAhdJnhfffdd/ik6GmnncY3v/nN\nQa2nx+VzI+Jc4LOZ+efV7c8AZObnq9vjgB3Ab6pP+UPgV8D8zDzi+rgunyupHi6f29jlczcA0yKi\nNSKOA64GVh06mJmvZ+bJmTk1M6cCP6CHMJckNV6PgZ6ZncAS4AlgC/BQZm6KiNsjYn5/FyhJqk9d\nc+iZuRpY3WXfrUdoe1Hfy5Kk38pMImKwyxhQx/Lb5Lz1X9KQ1tzczJ49e44p4IarzGTPnj00Nzf3\n6nne+i9pSGtpaaGjo4ORdqlzc3MzLS0tvXqOgS5pSBszZgytra2DXcaw4JSLJBXCQJekQhjoklQI\nA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQ\nJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12S\nCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpELUFegRMTcitkbE9oi4pZvjiyNiY0S8EBHrImJG\n40uVJB1Nj4EeEaOAe4B5wAxgYTeB/a3MnJmZs4AvAF9qeKWSpKOqZ4R+NrA9M3dm5gHgQWBBbYPM\n/HXN5glANq5ESVI9RtfRZjLwSs12B3BO10YRcQPwaeA44OKGVCdJqlvDTopm5j2ZeTrwd8A/dtcm\nIq6PiLaIaNu9e3ej3lqSRH2B/iowpWa7pbrvSB4EPtbdgcxclplzMnPOxIkT669SktSjegJ9AzAt\nIloj4jjgamBVbYOImFazeQmwrXElSpLq0eMcemZ2RsQS4AlgFLA8MzdFxO1AW2auApZExIeBt4DX\ngGv7s2hJ0u+r56QombkaWN1l3601j29scF2SpF7yTlFJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJU\nCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw\n0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANd\nkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVIi6Aj0i5kbE1ojYHhG3dHP80xGxOSJe\niognI+LUxpcqSTqaHgM9IkYB9wDzgBnAwoiY0aXZ88CczPwA8G3gC40uVJJ0dPWM0M8Gtmfmzsw8\nADwILKhtkJlrM3NfdfMHQEtjy5Qk9aSeQJ8MvFKz3VHddySfBP67uwMRcX1EtEVE2+7du+uvUpLU\no4aeFI2Ia4A5wBe7O56ZyzJzTmbOmThxYiPfWpJGvNF1tHkVmFKz3VLd9zsi4sPAPwAXZuabjSlP\nklSvekboG4BpEdEaEccBVwOrahtExJnA14D5mbmr8WVKknrSY6BnZiewBHgC2AI8lJmbIuL2iJhf\nbfZF4J3Af0TECxGx6ggvJ0nqJ/VMuZCZq4HVXfbdWvP4ww2uS5LUS94pKkmFMNAlqRAGuiQVwkCX\npEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVIi6Vlscatbv3NOr9ud+\nqJ8KkaQhxBG6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJU\nCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw\n0CWpEHUFekTMjYitEbE9Im7p5vifRsSPIqIzIi5vfJmSpJ70GOgRMQq4B5gHzAAWRsSMLs1+BnwC\n+FajC5Qk1Wd0HW3OBrZn5k6AiHgQWABsPtQgM9urxw72Q42SpDrUM+UyGXilZrujuq/XIuL6iGiL\niLbdu3cfy0tIko5gQE+KZuayzJyTmXMmTpw4kG8tScWrJ9BfBabUbLdU90mShpB6An0DMC0iWiPi\nOOBqYFX/liVJ6q0eAz0zO4ElwBPAFuChzNwUEbdHxHyAiPiTiOgArgC+FhGb+rNoSdLvq+cqFzJz\nNbC6y75bax5voDIVI0kaJN4pKkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSpEXTcW\nSYNh6Zqf9Kr9TR95Tz9VIg0PjtAlqRCO0HXMHEFLQ4sjdEkqhIEuSYUw0CWpEAa6JBXCk6IaML09\niSqpdxyhS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEJ42WKDuK6JpMHmCF2SCmGgS1IhDHRJKoRz\n6DrMW/Ol4c0RuiQVwkCXpEIY6JJUCANdkgphoEtSIbzKpUE++LNlvXzGnf1Sh6SRyxG6JBViRIzQ\nj+X6atdakTTcjIhAPxa9/SHwwX6qQ5Lq5ZSLJBViRIzQe3/CUpKGn7oCPSLmAv8GjAK+npn/0uX4\n8cAK4CxgD3BVZrY3ttSRzXVWJPWkx0CPiFHAPcBHgA5gQ0SsyszNNc0+CbyWmWdExNXAHcBV/VFw\nKQzoxivhe+rJePVFPSP0s4HtmbkTICIeBBYAtYG+APhs9fG3gbsjIjIzG1hrUZwGarwfvPv6XrUf\nin8HS9f0rg+91ds+9/Z7OhT1d5+P5XN07if75z6U6ClzI+JyYG5m/lV1exFwTmYuqWnz42qbjur2\njmqbX3Z5reuBQ9+tPwK2NqojXZwM/LLHVuUYaf0F+zwSjLT+Qn19PjUzJ3Z3YEBPimbmMqDfh0UR\n0ZaZc/r7fYaKkdZfsM8jwUjrL/S9z/VctvgqMKVmu6W6r9s2ETEaGEfl5KgkaYDUE+gbgGkR0RoR\nxwFXA6u6tFkFXFt9fDnwlPPnkjSwepxyyczOiFgCPEHlssXlmbkpIm4H2jJzFfANYGVEbAd+RSX0\nB9PQO9vVv0Zaf8E+jwQjrb/Qxz73eFJUkjQ8eOu/JBXCQJekQhQV6BExNyK2RsT2iLhlsOvpDxGx\nPCJ2Va/9P7TvXRGxJiK2Vb+OH8waGykipkTE2ojYHBGbIuLG6v6S+9wcEf8bES9W+3xbdX9rRPyw\n+vn+9+pFCsWIiFER8XxEPFbdLr2/7RGxMSJeiIi26r4+fa6LCfSaJQrmATOAhRExY3Cr6hf3AnO7\n7LsFeDIzpwFPVrdL0QncnJkzqKxSfEP177XkPr8JXJyZfwzMAuZGxAepLKmxNDPPAF6jsuRGSW4E\nttRsl95fgA9l5qyaa8/79LkuJtCpWaIgMw8Ah5YoKEpmPk3lSqJaC4D7qo/vAz42oEX1o8z8eWb+\nqPr4DSr/4CdTdp8zM39T3RxT/ZPAxVSW1oDC+hwRLcAlwNer20HB/T2KPn2uSwr0ycArNdsd1X0j\nwaTM/Hn18f8BkwazmP4SEVOBM4EfUnifq9MPLwC7gDXADmBvZnZWm5T2+f5X4G+Bg9XtCZTdX6j8\nkP6fiHiuuiwK9PFzPSLWQx9JMjMjorhrUSPincDDwF9n5q8rA7iKEvucmW8DsyLiJOA7wPRBLqnf\nRMSlwK7MfC4iLhrsegbQ+Zn5akT8AbAmIl6uPXgsn+uSRuj1LFFQql9ExCkA1a+7BrmehoqIMVTC\n/P7MfKS6u+g+H5KZe4G1wLnASdWlNaCsz/d5wPyIaKcyVXoxld+/UGp/AcjMV6tfd1H5oX02ffxc\nlxTo9SxRUKrapReuBR4dxFoaqjqX+g1gS2Z+qeZQyX2eWB2ZExFjqfwugi1Ugv3yarNi+pyZn8nM\nlsycSuXf7VOZ+ZcU2l+AiDghIk489Bj4M+DH9PFzXdSdohHxF1Tm4g4tUfC5QS6p4SLiAeAiKsts\n/gL4J+A/gYeAdwM/Ba7MzK4nToeliDgfeAbYyG/nV/+eyjx6qX3+AJUTYqOoDLoeyszbI+I0KiPY\ndwHPA9dk5puDV2njVadc/iYzLy25v9W+fae6ORr4VmZ+LiIm0IfPdVGBLkkjWUlTLpI0ohnoklQI\nA12SCmGgS1IhDHRJKoSBLkmFMNAlqRD/D69Vy/6A7istAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_4Rwgne7Gvwh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_x[train_]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YPL55OG8Glek",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        },
        "outputId": "eee752ac-4db0-491f-df5e-7e57f2b74824"
      },
      "source": [
        "train_x.delta_queue_time_oper_start_time.groupby(train_y).describe()"
      ],
      "execution_count": 471,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "      <th>mean</th>\n",
              "      <th>std</th>\n",
              "      <th>min</th>\n",
              "      <th>25%</th>\n",
              "      <th>50%</th>\n",
              "      <th>75%</th>\n",
              "      <th>max</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Метка</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>13537.0</td>\n",
              "      <td>9.258104</td>\n",
              "      <td>51.151799</td>\n",
              "      <td>-1.2345</td>\n",
              "      <td>-1.2345</td>\n",
              "      <td>-1.2345</td>\n",
              "      <td>0.0</td>\n",
              "      <td>806.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>8025.0</td>\n",
              "      <td>28.230901</td>\n",
              "      <td>65.880582</td>\n",
              "      <td>-1.2345</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>15.0000</td>\n",
              "      <td>19.0</td>\n",
              "      <td>825.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         count       mean        std     min     25%      50%   75%    max\n",
              "Метка                                                                     \n",
              "0      13537.0   9.258104  51.151799 -1.2345 -1.2345  -1.2345   0.0  806.0\n",
              "1       8025.0  28.230901  65.880582 -1.2345  0.0000  15.0000  19.0  825.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 471
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PpabhegVg3zT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "08a45c8b-a0bd-4da4-d738-c1439178a122"
      },
      "source": [
        "\n",
        "train_x.isnull().mean().mean(), test_x.isnull().mean().mean(), train_x.shape, test_x.shape"
      ],
      "execution_count": 311,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.0, 0.0, (21562, 53), (6469, 53))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 311
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k7lRav5NgbmP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from catboost import CatBoostClassifier, Pool"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_kNbYY7lz_M0",
        "colab_type": "text"
      },
      "source": [
        "Неудачный эксперимент - фича из линейного классификатора. Профита особо не дала. \n",
        "\n",
        "```\n",
        "from sklearn.model_selection import cross_val_predict\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "lr = LogisticRegression(max_iter=10000, solver='lbfgs')\n",
        "lr.fit(train_x, train_y)\n",
        "print(f1_score(test_y, lr.predict(test_x)))\n",
        "\n",
        "for t in (train_x, test_x, valid_x):\n",
        "    if 'logreg_score' in test_x.columns:\n",
        "        t.drop('logreg_score', inplace=True, axis=1)\n",
        "logreg_train = cross_val_predict(lr, train_x, train_y, method='predict_proba')[:, 1]\n",
        "train_x['logreg_score'] = logreg_train\n",
        "test_x['logreg_score'] = lr.predict_proba(test_x)[:, 1]\n",
        "valid_x['logreg_score'] = lr.predict_proba(valid_x)[:, 1]\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fyjJH1zr6tzx",
        "colab_type": "text"
      },
      "source": [
        "Ещё один неудачный эксперимент - PCA-фичи\n",
        "```\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.pipeline import make_pipeline\n",
        "\n",
        "n_comp = 10\n",
        "pca_colnames = ['pca_{}'.format(i) for i in range(n_comp)]\n",
        "\n",
        "pca = make_pipeline(\n",
        "    StandardScaler(),\n",
        "    PCA(n_components=n_comp),\n",
        ")\n",
        "\n",
        "for t in (train_x, test_x, valid_x):\n",
        "    if 'pca_0' in test_x.columns:\n",
        "        t.drop(pca_colnames, inplace=True, axis=1)\n",
        "\n",
        "pca.fit(train_x)\n",
        "\n",
        "\n",
        "for t in (train_x, test_x, valid_x):\n",
        "    pca_df = pd.DataFrame(pca.transform(t), columns=pca_colnames)\n",
        "    for c in pca_colnames:\n",
        "        t[c] = 0\n",
        "    t[pca_colnames] = pca_df\n",
        "\n",
        "plt.plot(np.cumsum(pca.steps[-1][1].explained_variance_ratio_))\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MD8MVwR93nP3",
        "colab_type": "text"
      },
      "source": [
        "## обучаем модель и подбираем параметры"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UWD8d9xkhMYg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "240b9e8c-62b6-4279-c873-3cf2ba1e6efa"
      },
      "source": [
        "eval_set = Pool(data=test_x, label=test_y)\n",
        "clf = CatBoostClassifier(\n",
        "    n_estimators=1000, learning_rate=0.005, \n",
        "    use_best_model=True, eval_metric='AUC', random_seed=1,\n",
        "    grow_policy='SymmetricTree', depth=6,\n",
        "    rsm=0.5,\n",
        ")\n",
        "clf.fit(train_x, train_y, eval_set=eval_set)"
      ],
      "execution_count": 592,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0:\ttest: 0.9242489\tbest: 0.9242489 (0)\ttotal: 27.6ms\tremaining: 27.5s\n",
            "1:\ttest: 0.9307170\tbest: 0.9307170 (1)\ttotal: 53.3ms\tremaining: 26.6s\n",
            "2:\ttest: 0.9338353\tbest: 0.9338353 (2)\ttotal: 77.4ms\tremaining: 25.7s\n",
            "3:\ttest: 0.9348804\tbest: 0.9348804 (3)\ttotal: 103ms\tremaining: 25.8s\n",
            "4:\ttest: 0.9366899\tbest: 0.9366899 (4)\ttotal: 128ms\tremaining: 25.5s\n",
            "5:\ttest: 0.9375189\tbest: 0.9375189 (5)\ttotal: 154ms\tremaining: 25.4s\n",
            "6:\ttest: 0.9374581\tbest: 0.9375189 (5)\ttotal: 180ms\tremaining: 25.5s\n",
            "7:\ttest: 0.9376407\tbest: 0.9376407 (7)\ttotal: 205ms\tremaining: 25.4s\n",
            "8:\ttest: 0.9374992\tbest: 0.9376407 (7)\ttotal: 237ms\tremaining: 26.1s\n",
            "9:\ttest: 0.9373343\tbest: 0.9376407 (7)\ttotal: 263ms\tremaining: 26s\n",
            "10:\ttest: 0.9373881\tbest: 0.9376407 (7)\ttotal: 288ms\tremaining: 25.9s\n",
            "11:\ttest: 0.9375298\tbest: 0.9376407 (7)\ttotal: 313ms\tremaining: 25.8s\n",
            "12:\ttest: 0.9375353\tbest: 0.9376407 (7)\ttotal: 339ms\tremaining: 25.7s\n",
            "13:\ttest: 0.9379193\tbest: 0.9379193 (13)\ttotal: 365ms\tremaining: 25.7s\n",
            "14:\ttest: 0.9378828\tbest: 0.9379193 (13)\ttotal: 391ms\tremaining: 25.6s\n",
            "15:\ttest: 0.9378704\tbest: 0.9379193 (13)\ttotal: 417ms\tremaining: 25.6s\n",
            "16:\ttest: 0.9378124\tbest: 0.9379193 (13)\ttotal: 448ms\tremaining: 25.9s\n",
            "17:\ttest: 0.9380384\tbest: 0.9380384 (17)\ttotal: 474ms\tremaining: 25.8s\n",
            "18:\ttest: 0.9379451\tbest: 0.9380384 (17)\ttotal: 499ms\tremaining: 25.8s\n",
            "19:\ttest: 0.9378161\tbest: 0.9380384 (17)\ttotal: 524ms\tremaining: 25.7s\n",
            "20:\ttest: 0.9379800\tbest: 0.9380384 (17)\ttotal: 551ms\tremaining: 25.7s\n",
            "21:\ttest: 0.9380927\tbest: 0.9380927 (21)\ttotal: 576ms\tremaining: 25.6s\n",
            "22:\ttest: 0.9382589\tbest: 0.9382589 (22)\ttotal: 600ms\tremaining: 25.5s\n",
            "23:\ttest: 0.9382417\tbest: 0.9382589 (22)\ttotal: 625ms\tremaining: 25.4s\n",
            "24:\ttest: 0.9382357\tbest: 0.9382589 (22)\ttotal: 655ms\tremaining: 25.5s\n",
            "25:\ttest: 0.9382445\tbest: 0.9382589 (22)\ttotal: 687ms\tremaining: 25.7s\n",
            "26:\ttest: 0.9382396\tbest: 0.9382589 (22)\ttotal: 712ms\tremaining: 25.7s\n",
            "27:\ttest: 0.9382650\tbest: 0.9382650 (27)\ttotal: 736ms\tremaining: 25.5s\n",
            "28:\ttest: 0.9383703\tbest: 0.9383703 (28)\ttotal: 759ms\tremaining: 25.4s\n",
            "29:\ttest: 0.9382809\tbest: 0.9383703 (28)\ttotal: 783ms\tremaining: 25.3s\n",
            "30:\ttest: 0.9383701\tbest: 0.9383703 (28)\ttotal: 809ms\tremaining: 25.3s\n",
            "31:\ttest: 0.9384162\tbest: 0.9384162 (31)\ttotal: 834ms\tremaining: 25.2s\n",
            "32:\ttest: 0.9385881\tbest: 0.9385881 (32)\ttotal: 866ms\tremaining: 25.4s\n",
            "33:\ttest: 0.9386081\tbest: 0.9386081 (33)\ttotal: 890ms\tremaining: 25.3s\n",
            "34:\ttest: 0.9381471\tbest: 0.9386081 (33)\ttotal: 914ms\tremaining: 25.2s\n",
            "35:\ttest: 0.9381112\tbest: 0.9386081 (33)\ttotal: 940ms\tremaining: 25.2s\n",
            "36:\ttest: 0.9381094\tbest: 0.9386081 (33)\ttotal: 965ms\tremaining: 25.1s\n",
            "37:\ttest: 0.9381682\tbest: 0.9386081 (33)\ttotal: 990ms\tremaining: 25.1s\n",
            "38:\ttest: 0.9382984\tbest: 0.9386081 (33)\ttotal: 1.01s\tremaining: 25s\n",
            "39:\ttest: 0.9384696\tbest: 0.9386081 (33)\ttotal: 1.04s\tremaining: 25s\n",
            "40:\ttest: 0.9385787\tbest: 0.9386081 (33)\ttotal: 1.07s\tremaining: 25s\n",
            "41:\ttest: 0.9386687\tbest: 0.9386687 (41)\ttotal: 1.09s\tremaining: 25s\n",
            "42:\ttest: 0.9387260\tbest: 0.9387260 (42)\ttotal: 1.12s\tremaining: 24.9s\n",
            "43:\ttest: 0.9386652\tbest: 0.9387260 (42)\ttotal: 1.14s\tremaining: 24.9s\n",
            "44:\ttest: 0.9388907\tbest: 0.9388907 (44)\ttotal: 1.17s\tremaining: 24.8s\n",
            "45:\ttest: 0.9387909\tbest: 0.9388907 (44)\ttotal: 1.19s\tremaining: 24.8s\n",
            "46:\ttest: 0.9388396\tbest: 0.9388907 (44)\ttotal: 1.22s\tremaining: 24.7s\n",
            "47:\ttest: 0.9391637\tbest: 0.9391637 (47)\ttotal: 1.24s\tremaining: 24.7s\n",
            "48:\ttest: 0.9392351\tbest: 0.9392351 (48)\ttotal: 1.27s\tremaining: 24.6s\n",
            "49:\ttest: 0.9390689\tbest: 0.9392351 (48)\ttotal: 1.3s\tremaining: 24.6s\n",
            "50:\ttest: 0.9390790\tbest: 0.9392351 (48)\ttotal: 1.32s\tremaining: 24.6s\n",
            "51:\ttest: 0.9391315\tbest: 0.9392351 (48)\ttotal: 1.35s\tremaining: 24.6s\n",
            "52:\ttest: 0.9391450\tbest: 0.9392351 (48)\ttotal: 1.37s\tremaining: 24.6s\n",
            "53:\ttest: 0.9392419\tbest: 0.9392419 (53)\ttotal: 1.4s\tremaining: 24.5s\n",
            "54:\ttest: 0.9391437\tbest: 0.9392419 (53)\ttotal: 1.42s\tremaining: 24.5s\n",
            "55:\ttest: 0.9390841\tbest: 0.9392419 (53)\ttotal: 1.45s\tremaining: 24.5s\n",
            "56:\ttest: 0.9390855\tbest: 0.9392419 (53)\ttotal: 1.48s\tremaining: 24.5s\n",
            "57:\ttest: 0.9391370\tbest: 0.9392419 (53)\ttotal: 1.51s\tremaining: 24.6s\n",
            "58:\ttest: 0.9391456\tbest: 0.9392419 (53)\ttotal: 1.54s\tremaining: 24.5s\n",
            "59:\ttest: 0.9392120\tbest: 0.9392419 (53)\ttotal: 1.56s\tremaining: 24.5s\n",
            "60:\ttest: 0.9392100\tbest: 0.9392419 (53)\ttotal: 1.59s\tremaining: 24.5s\n",
            "61:\ttest: 0.9392166\tbest: 0.9392419 (53)\ttotal: 1.62s\tremaining: 24.5s\n",
            "62:\ttest: 0.9390799\tbest: 0.9392419 (53)\ttotal: 1.64s\tremaining: 24.4s\n",
            "63:\ttest: 0.9390743\tbest: 0.9392419 (53)\ttotal: 1.67s\tremaining: 24.4s\n",
            "64:\ttest: 0.9390607\tbest: 0.9392419 (53)\ttotal: 1.73s\tremaining: 24.8s\n",
            "65:\ttest: 0.9390375\tbest: 0.9392419 (53)\ttotal: 1.75s\tremaining: 24.8s\n",
            "66:\ttest: 0.9390081\tbest: 0.9392419 (53)\ttotal: 1.78s\tremaining: 24.8s\n",
            "67:\ttest: 0.9391278\tbest: 0.9392419 (53)\ttotal: 1.8s\tremaining: 24.7s\n",
            "68:\ttest: 0.9391259\tbest: 0.9392419 (53)\ttotal: 1.83s\tremaining: 24.7s\n",
            "69:\ttest: 0.9392433\tbest: 0.9392433 (69)\ttotal: 1.86s\tremaining: 24.7s\n",
            "70:\ttest: 0.9392385\tbest: 0.9392433 (69)\ttotal: 1.88s\tremaining: 24.6s\n",
            "71:\ttest: 0.9392554\tbest: 0.9392554 (71)\ttotal: 1.91s\tremaining: 24.6s\n",
            "72:\ttest: 0.9392676\tbest: 0.9392676 (72)\ttotal: 1.94s\tremaining: 24.6s\n",
            "73:\ttest: 0.9393244\tbest: 0.9393244 (73)\ttotal: 1.96s\tremaining: 24.6s\n",
            "74:\ttest: 0.9392888\tbest: 0.9393244 (73)\ttotal: 1.99s\tremaining: 24.5s\n",
            "75:\ttest: 0.9392476\tbest: 0.9393244 (73)\ttotal: 2.01s\tremaining: 24.5s\n",
            "76:\ttest: 0.9392712\tbest: 0.9393244 (73)\ttotal: 2.04s\tremaining: 24.4s\n",
            "77:\ttest: 0.9393965\tbest: 0.9393965 (77)\ttotal: 2.07s\tremaining: 24.5s\n",
            "78:\ttest: 0.9393288\tbest: 0.9393965 (77)\ttotal: 2.1s\tremaining: 24.5s\n",
            "79:\ttest: 0.9393359\tbest: 0.9393965 (77)\ttotal: 2.13s\tremaining: 24.4s\n",
            "80:\ttest: 0.9393887\tbest: 0.9393965 (77)\ttotal: 2.16s\tremaining: 24.5s\n",
            "81:\ttest: 0.9394167\tbest: 0.9394167 (81)\ttotal: 2.18s\tremaining: 24.4s\n",
            "82:\ttest: 0.9394656\tbest: 0.9394656 (82)\ttotal: 2.21s\tremaining: 24.4s\n",
            "83:\ttest: 0.9394827\tbest: 0.9394827 (83)\ttotal: 2.23s\tremaining: 24.3s\n",
            "84:\ttest: 0.9393973\tbest: 0.9394827 (83)\ttotal: 2.26s\tremaining: 24.4s\n",
            "85:\ttest: 0.9394583\tbest: 0.9394827 (83)\ttotal: 2.29s\tremaining: 24.3s\n",
            "86:\ttest: 0.9394698\tbest: 0.9394827 (83)\ttotal: 2.31s\tremaining: 24.3s\n",
            "87:\ttest: 0.9394326\tbest: 0.9394827 (83)\ttotal: 2.34s\tremaining: 24.2s\n",
            "88:\ttest: 0.9394752\tbest: 0.9394827 (83)\ttotal: 2.37s\tremaining: 24.2s\n",
            "89:\ttest: 0.9394713\tbest: 0.9394827 (83)\ttotal: 2.4s\tremaining: 24.3s\n",
            "90:\ttest: 0.9394783\tbest: 0.9394827 (83)\ttotal: 2.42s\tremaining: 24.2s\n",
            "91:\ttest: 0.9394986\tbest: 0.9394986 (91)\ttotal: 2.45s\tremaining: 24.2s\n",
            "92:\ttest: 0.9394943\tbest: 0.9394986 (91)\ttotal: 2.47s\tremaining: 24.1s\n",
            "93:\ttest: 0.9394753\tbest: 0.9394986 (91)\ttotal: 2.5s\tremaining: 24.1s\n",
            "94:\ttest: 0.9394546\tbest: 0.9394986 (91)\ttotal: 2.53s\tremaining: 24.1s\n",
            "95:\ttest: 0.9394845\tbest: 0.9394986 (91)\ttotal: 2.55s\tremaining: 24s\n",
            "96:\ttest: 0.9394787\tbest: 0.9394986 (91)\ttotal: 2.58s\tremaining: 24s\n",
            "97:\ttest: 0.9394160\tbest: 0.9394986 (91)\ttotal: 2.61s\tremaining: 24s\n",
            "98:\ttest: 0.9394433\tbest: 0.9394986 (91)\ttotal: 2.63s\tremaining: 24s\n",
            "99:\ttest: 0.9393321\tbest: 0.9394986 (91)\ttotal: 2.66s\tremaining: 23.9s\n",
            "100:\ttest: 0.9394453\tbest: 0.9394986 (91)\ttotal: 2.68s\tremaining: 23.9s\n",
            "101:\ttest: 0.9394009\tbest: 0.9394986 (91)\ttotal: 2.71s\tremaining: 23.9s\n",
            "102:\ttest: 0.9394188\tbest: 0.9394986 (91)\ttotal: 2.73s\tremaining: 23.8s\n",
            "103:\ttest: 0.9394122\tbest: 0.9394986 (91)\ttotal: 2.76s\tremaining: 23.8s\n",
            "104:\ttest: 0.9394229\tbest: 0.9394986 (91)\ttotal: 2.79s\tremaining: 23.8s\n",
            "105:\ttest: 0.9393914\tbest: 0.9394986 (91)\ttotal: 2.82s\tremaining: 23.8s\n",
            "106:\ttest: 0.9394325\tbest: 0.9394986 (91)\ttotal: 2.84s\tremaining: 23.7s\n",
            "107:\ttest: 0.9394170\tbest: 0.9394986 (91)\ttotal: 2.87s\tremaining: 23.7s\n",
            "108:\ttest: 0.9393907\tbest: 0.9394986 (91)\ttotal: 2.89s\tremaining: 23.6s\n",
            "109:\ttest: 0.9394298\tbest: 0.9394986 (91)\ttotal: 2.92s\tremaining: 23.6s\n",
            "110:\ttest: 0.9394477\tbest: 0.9394986 (91)\ttotal: 2.94s\tremaining: 23.6s\n",
            "111:\ttest: 0.9394189\tbest: 0.9394986 (91)\ttotal: 2.97s\tremaining: 23.6s\n",
            "112:\ttest: 0.9393222\tbest: 0.9394986 (91)\ttotal: 3s\tremaining: 23.5s\n",
            "113:\ttest: 0.9393152\tbest: 0.9394986 (91)\ttotal: 3.02s\tremaining: 23.5s\n",
            "114:\ttest: 0.9393298\tbest: 0.9394986 (91)\ttotal: 3.05s\tremaining: 23.5s\n",
            "115:\ttest: 0.9393089\tbest: 0.9394986 (91)\ttotal: 3.07s\tremaining: 23.4s\n",
            "116:\ttest: 0.9393028\tbest: 0.9394986 (91)\ttotal: 3.1s\tremaining: 23.4s\n",
            "117:\ttest: 0.9393256\tbest: 0.9394986 (91)\ttotal: 3.13s\tremaining: 23.4s\n",
            "118:\ttest: 0.9393613\tbest: 0.9394986 (91)\ttotal: 3.15s\tremaining: 23.4s\n",
            "119:\ttest: 0.9393418\tbest: 0.9394986 (91)\ttotal: 3.18s\tremaining: 23.3s\n",
            "120:\ttest: 0.9393992\tbest: 0.9394986 (91)\ttotal: 3.21s\tremaining: 23.3s\n",
            "121:\ttest: 0.9393905\tbest: 0.9394986 (91)\ttotal: 3.24s\tremaining: 23.3s\n",
            "122:\ttest: 0.9393791\tbest: 0.9394986 (91)\ttotal: 3.27s\tremaining: 23.3s\n",
            "123:\ttest: 0.9394289\tbest: 0.9394986 (91)\ttotal: 3.29s\tremaining: 23.3s\n",
            "124:\ttest: 0.9394425\tbest: 0.9394986 (91)\ttotal: 3.32s\tremaining: 23.2s\n",
            "125:\ttest: 0.9394528\tbest: 0.9394986 (91)\ttotal: 3.34s\tremaining: 23.2s\n",
            "126:\ttest: 0.9394132\tbest: 0.9394986 (91)\ttotal: 3.37s\tremaining: 23.1s\n",
            "127:\ttest: 0.9394185\tbest: 0.9394986 (91)\ttotal: 3.39s\tremaining: 23.1s\n",
            "128:\ttest: 0.9394268\tbest: 0.9394986 (91)\ttotal: 3.42s\tremaining: 23.1s\n",
            "129:\ttest: 0.9394660\tbest: 0.9394986 (91)\ttotal: 3.45s\tremaining: 23.1s\n",
            "130:\ttest: 0.9394491\tbest: 0.9394986 (91)\ttotal: 3.47s\tremaining: 23s\n",
            "131:\ttest: 0.9394561\tbest: 0.9394986 (91)\ttotal: 3.5s\tremaining: 23s\n",
            "132:\ttest: 0.9394783\tbest: 0.9394986 (91)\ttotal: 3.53s\tremaining: 23s\n",
            "133:\ttest: 0.9394775\tbest: 0.9394986 (91)\ttotal: 3.55s\tremaining: 23s\n",
            "134:\ttest: 0.9395191\tbest: 0.9395191 (134)\ttotal: 3.58s\tremaining: 22.9s\n",
            "135:\ttest: 0.9395574\tbest: 0.9395574 (135)\ttotal: 3.6s\tremaining: 22.9s\n",
            "136:\ttest: 0.9395653\tbest: 0.9395653 (136)\ttotal: 3.63s\tremaining: 22.9s\n",
            "137:\ttest: 0.9395395\tbest: 0.9395653 (136)\ttotal: 3.66s\tremaining: 22.9s\n",
            "138:\ttest: 0.9395658\tbest: 0.9395658 (138)\ttotal: 3.68s\tremaining: 22.8s\n",
            "139:\ttest: 0.9396175\tbest: 0.9396175 (139)\ttotal: 3.71s\tremaining: 22.8s\n",
            "140:\ttest: 0.9395618\tbest: 0.9396175 (139)\ttotal: 3.73s\tremaining: 22.7s\n",
            "141:\ttest: 0.9395808\tbest: 0.9396175 (139)\ttotal: 3.76s\tremaining: 22.7s\n",
            "142:\ttest: 0.9395802\tbest: 0.9396175 (139)\ttotal: 3.78s\tremaining: 22.7s\n",
            "143:\ttest: 0.9396072\tbest: 0.9396175 (139)\ttotal: 3.81s\tremaining: 22.6s\n",
            "144:\ttest: 0.9396509\tbest: 0.9396509 (144)\ttotal: 3.84s\tremaining: 22.6s\n",
            "145:\ttest: 0.9396785\tbest: 0.9396785 (145)\ttotal: 3.86s\tremaining: 22.6s\n",
            "146:\ttest: 0.9397156\tbest: 0.9397156 (146)\ttotal: 3.89s\tremaining: 22.6s\n",
            "147:\ttest: 0.9397337\tbest: 0.9397337 (147)\ttotal: 3.91s\tremaining: 22.5s\n",
            "148:\ttest: 0.9397298\tbest: 0.9397337 (147)\ttotal: 3.94s\tremaining: 22.5s\n",
            "149:\ttest: 0.9397463\tbest: 0.9397463 (149)\ttotal: 3.96s\tremaining: 22.5s\n",
            "150:\ttest: 0.9397593\tbest: 0.9397593 (150)\ttotal: 3.99s\tremaining: 22.4s\n",
            "151:\ttest: 0.9397261\tbest: 0.9397593 (150)\ttotal: 4.01s\tremaining: 22.4s\n",
            "152:\ttest: 0.9396981\tbest: 0.9397593 (150)\ttotal: 4.05s\tremaining: 22.4s\n",
            "153:\ttest: 0.9397053\tbest: 0.9397593 (150)\ttotal: 4.08s\tremaining: 22.4s\n",
            "154:\ttest: 0.9396812\tbest: 0.9397593 (150)\ttotal: 4.1s\tremaining: 22.4s\n",
            "155:\ttest: 0.9396628\tbest: 0.9397593 (150)\ttotal: 4.13s\tremaining: 22.3s\n",
            "156:\ttest: 0.9396727\tbest: 0.9397593 (150)\ttotal: 4.15s\tremaining: 22.3s\n",
            "157:\ttest: 0.9396647\tbest: 0.9397593 (150)\ttotal: 4.17s\tremaining: 22.2s\n",
            "158:\ttest: 0.9396723\tbest: 0.9397593 (150)\ttotal: 4.19s\tremaining: 22.2s\n",
            "159:\ttest: 0.9396608\tbest: 0.9397593 (150)\ttotal: 4.22s\tremaining: 22.1s\n",
            "160:\ttest: 0.9396987\tbest: 0.9397593 (150)\ttotal: 4.24s\tremaining: 22.1s\n",
            "161:\ttest: 0.9396914\tbest: 0.9397593 (150)\ttotal: 4.26s\tremaining: 22.1s\n",
            "162:\ttest: 0.9396563\tbest: 0.9397593 (150)\ttotal: 4.29s\tremaining: 22s\n",
            "163:\ttest: 0.9396437\tbest: 0.9397593 (150)\ttotal: 4.31s\tremaining: 22s\n",
            "164:\ttest: 0.9397113\tbest: 0.9397593 (150)\ttotal: 4.33s\tremaining: 21.9s\n",
            "165:\ttest: 0.9397235\tbest: 0.9397593 (150)\ttotal: 4.35s\tremaining: 21.9s\n",
            "166:\ttest: 0.9396781\tbest: 0.9397593 (150)\ttotal: 4.38s\tremaining: 21.8s\n",
            "167:\ttest: 0.9396830\tbest: 0.9397593 (150)\ttotal: 4.4s\tremaining: 21.8s\n",
            "168:\ttest: 0.9396556\tbest: 0.9397593 (150)\ttotal: 4.43s\tremaining: 21.8s\n",
            "169:\ttest: 0.9396764\tbest: 0.9397593 (150)\ttotal: 4.45s\tremaining: 21.7s\n",
            "170:\ttest: 0.9396602\tbest: 0.9397593 (150)\ttotal: 4.48s\tremaining: 21.7s\n",
            "171:\ttest: 0.9396685\tbest: 0.9397593 (150)\ttotal: 4.5s\tremaining: 21.7s\n",
            "172:\ttest: 0.9396589\tbest: 0.9397593 (150)\ttotal: 4.52s\tremaining: 21.6s\n",
            "173:\ttest: 0.9396539\tbest: 0.9397593 (150)\ttotal: 4.54s\tremaining: 21.6s\n",
            "174:\ttest: 0.9396031\tbest: 0.9397593 (150)\ttotal: 4.57s\tremaining: 21.5s\n",
            "175:\ttest: 0.9395693\tbest: 0.9397593 (150)\ttotal: 4.59s\tremaining: 21.5s\n",
            "176:\ttest: 0.9395722\tbest: 0.9397593 (150)\ttotal: 4.61s\tremaining: 21.4s\n",
            "177:\ttest: 0.9395809\tbest: 0.9397593 (150)\ttotal: 4.63s\tremaining: 21.4s\n",
            "178:\ttest: 0.9395827\tbest: 0.9397593 (150)\ttotal: 4.66s\tremaining: 21.4s\n",
            "179:\ttest: 0.9395839\tbest: 0.9397593 (150)\ttotal: 4.68s\tremaining: 21.3s\n",
            "180:\ttest: 0.9395750\tbest: 0.9397593 (150)\ttotal: 4.71s\tremaining: 21.3s\n",
            "181:\ttest: 0.9395678\tbest: 0.9397593 (150)\ttotal: 4.74s\tremaining: 21.3s\n",
            "182:\ttest: 0.9395923\tbest: 0.9397593 (150)\ttotal: 4.76s\tremaining: 21.2s\n",
            "183:\ttest: 0.9396040\tbest: 0.9397593 (150)\ttotal: 4.78s\tremaining: 21.2s\n",
            "184:\ttest: 0.9396013\tbest: 0.9397593 (150)\ttotal: 4.8s\tremaining: 21.1s\n",
            "185:\ttest: 0.9396015\tbest: 0.9397593 (150)\ttotal: 4.82s\tremaining: 21.1s\n",
            "186:\ttest: 0.9396067\tbest: 0.9397593 (150)\ttotal: 4.84s\tremaining: 21.1s\n",
            "187:\ttest: 0.9396263\tbest: 0.9397593 (150)\ttotal: 4.87s\tremaining: 21s\n",
            "188:\ttest: 0.9396420\tbest: 0.9397593 (150)\ttotal: 4.89s\tremaining: 21s\n",
            "189:\ttest: 0.9396372\tbest: 0.9397593 (150)\ttotal: 4.92s\tremaining: 21s\n",
            "190:\ttest: 0.9396443\tbest: 0.9397593 (150)\ttotal: 4.94s\tremaining: 20.9s\n",
            "191:\ttest: 0.9396583\tbest: 0.9397593 (150)\ttotal: 4.96s\tremaining: 20.9s\n",
            "192:\ttest: 0.9396698\tbest: 0.9397593 (150)\ttotal: 4.98s\tremaining: 20.8s\n",
            "193:\ttest: 0.9396753\tbest: 0.9397593 (150)\ttotal: 5.01s\tremaining: 20.8s\n",
            "194:\ttest: 0.9396602\tbest: 0.9397593 (150)\ttotal: 5.03s\tremaining: 20.8s\n",
            "195:\ttest: 0.9396754\tbest: 0.9397593 (150)\ttotal: 5.05s\tremaining: 20.7s\n",
            "196:\ttest: 0.9396898\tbest: 0.9397593 (150)\ttotal: 5.07s\tremaining: 20.7s\n",
            "197:\ttest: 0.9396908\tbest: 0.9397593 (150)\ttotal: 5.1s\tremaining: 20.7s\n",
            "198:\ttest: 0.9396953\tbest: 0.9397593 (150)\ttotal: 5.13s\tremaining: 20.6s\n",
            "199:\ttest: 0.9397077\tbest: 0.9397593 (150)\ttotal: 5.15s\tremaining: 20.6s\n",
            "200:\ttest: 0.9396909\tbest: 0.9397593 (150)\ttotal: 5.17s\tremaining: 20.6s\n",
            "201:\ttest: 0.9396922\tbest: 0.9397593 (150)\ttotal: 5.19s\tremaining: 20.5s\n",
            "202:\ttest: 0.9396842\tbest: 0.9397593 (150)\ttotal: 5.22s\tremaining: 20.5s\n",
            "203:\ttest: 0.9397104\tbest: 0.9397593 (150)\ttotal: 5.24s\tremaining: 20.4s\n",
            "204:\ttest: 0.9396962\tbest: 0.9397593 (150)\ttotal: 5.26s\tremaining: 20.4s\n",
            "205:\ttest: 0.9396959\tbest: 0.9397593 (150)\ttotal: 5.28s\tremaining: 20.4s\n",
            "206:\ttest: 0.9396774\tbest: 0.9397593 (150)\ttotal: 5.31s\tremaining: 20.3s\n",
            "207:\ttest: 0.9396730\tbest: 0.9397593 (150)\ttotal: 5.33s\tremaining: 20.3s\n",
            "208:\ttest: 0.9396771\tbest: 0.9397593 (150)\ttotal: 5.35s\tremaining: 20.3s\n",
            "209:\ttest: 0.9396868\tbest: 0.9397593 (150)\ttotal: 5.37s\tremaining: 20.2s\n",
            "210:\ttest: 0.9396831\tbest: 0.9397593 (150)\ttotal: 5.39s\tremaining: 20.2s\n",
            "211:\ttest: 0.9396924\tbest: 0.9397593 (150)\ttotal: 5.42s\tremaining: 20.1s\n",
            "212:\ttest: 0.9397040\tbest: 0.9397593 (150)\ttotal: 5.44s\tremaining: 20.1s\n",
            "213:\ttest: 0.9397060\tbest: 0.9397593 (150)\ttotal: 5.46s\tremaining: 20.1s\n",
            "214:\ttest: 0.9397145\tbest: 0.9397593 (150)\ttotal: 5.49s\tremaining: 20s\n",
            "215:\ttest: 0.9397258\tbest: 0.9397593 (150)\ttotal: 5.51s\tremaining: 20s\n",
            "216:\ttest: 0.9396908\tbest: 0.9397593 (150)\ttotal: 5.54s\tremaining: 20s\n",
            "217:\ttest: 0.9396950\tbest: 0.9397593 (150)\ttotal: 5.56s\tremaining: 19.9s\n",
            "218:\ttest: 0.9396947\tbest: 0.9397593 (150)\ttotal: 5.58s\tremaining: 19.9s\n",
            "219:\ttest: 0.9397276\tbest: 0.9397593 (150)\ttotal: 5.6s\tremaining: 19.9s\n",
            "220:\ttest: 0.9397195\tbest: 0.9397593 (150)\ttotal: 5.63s\tremaining: 19.8s\n",
            "221:\ttest: 0.9397155\tbest: 0.9397593 (150)\ttotal: 5.65s\tremaining: 19.8s\n",
            "222:\ttest: 0.9397084\tbest: 0.9397593 (150)\ttotal: 5.67s\tremaining: 19.8s\n",
            "223:\ttest: 0.9397165\tbest: 0.9397593 (150)\ttotal: 5.69s\tremaining: 19.7s\n",
            "224:\ttest: 0.9397311\tbest: 0.9397593 (150)\ttotal: 5.72s\tremaining: 19.7s\n",
            "225:\ttest: 0.9397462\tbest: 0.9397593 (150)\ttotal: 5.74s\tremaining: 19.7s\n",
            "226:\ttest: 0.9397533\tbest: 0.9397593 (150)\ttotal: 5.77s\tremaining: 19.6s\n",
            "227:\ttest: 0.9397567\tbest: 0.9397593 (150)\ttotal: 5.79s\tremaining: 19.6s\n",
            "228:\ttest: 0.9397618\tbest: 0.9397618 (228)\ttotal: 5.81s\tremaining: 19.6s\n",
            "229:\ttest: 0.9397620\tbest: 0.9397620 (229)\ttotal: 5.83s\tremaining: 19.5s\n",
            "230:\ttest: 0.9397658\tbest: 0.9397658 (230)\ttotal: 5.86s\tremaining: 19.5s\n",
            "231:\ttest: 0.9397736\tbest: 0.9397736 (231)\ttotal: 5.88s\tremaining: 19.5s\n",
            "232:\ttest: 0.9397703\tbest: 0.9397736 (231)\ttotal: 5.9s\tremaining: 19.4s\n",
            "233:\ttest: 0.9397441\tbest: 0.9397736 (231)\ttotal: 5.93s\tremaining: 19.4s\n",
            "234:\ttest: 0.9397383\tbest: 0.9397736 (231)\ttotal: 5.95s\tremaining: 19.4s\n",
            "235:\ttest: 0.9397248\tbest: 0.9397736 (231)\ttotal: 5.97s\tremaining: 19.3s\n",
            "236:\ttest: 0.9397249\tbest: 0.9397736 (231)\ttotal: 5.99s\tremaining: 19.3s\n",
            "237:\ttest: 0.9397246\tbest: 0.9397736 (231)\ttotal: 6.02s\tremaining: 19.3s\n",
            "238:\ttest: 0.9397466\tbest: 0.9397736 (231)\ttotal: 6.04s\tremaining: 19.2s\n",
            "239:\ttest: 0.9397560\tbest: 0.9397736 (231)\ttotal: 6.06s\tremaining: 19.2s\n",
            "240:\ttest: 0.9397385\tbest: 0.9397736 (231)\ttotal: 6.09s\tremaining: 19.2s\n",
            "241:\ttest: 0.9397393\tbest: 0.9397736 (231)\ttotal: 6.12s\tremaining: 19.2s\n",
            "242:\ttest: 0.9397409\tbest: 0.9397736 (231)\ttotal: 6.15s\tremaining: 19.2s\n",
            "243:\ttest: 0.9397520\tbest: 0.9397736 (231)\ttotal: 6.17s\tremaining: 19.1s\n",
            "244:\ttest: 0.9397518\tbest: 0.9397736 (231)\ttotal: 6.2s\tremaining: 19.1s\n",
            "245:\ttest: 0.9397573\tbest: 0.9397736 (231)\ttotal: 6.22s\tremaining: 19.1s\n",
            "246:\ttest: 0.9397653\tbest: 0.9397736 (231)\ttotal: 6.25s\tremaining: 19.1s\n",
            "247:\ttest: 0.9397702\tbest: 0.9397736 (231)\ttotal: 6.28s\tremaining: 19s\n",
            "248:\ttest: 0.9397664\tbest: 0.9397736 (231)\ttotal: 6.3s\tremaining: 19s\n",
            "249:\ttest: 0.9397581\tbest: 0.9397736 (231)\ttotal: 6.32s\tremaining: 19s\n",
            "250:\ttest: 0.9397702\tbest: 0.9397736 (231)\ttotal: 6.36s\tremaining: 19s\n",
            "251:\ttest: 0.9397633\tbest: 0.9397736 (231)\ttotal: 6.38s\tremaining: 18.9s\n",
            "252:\ttest: 0.9397617\tbest: 0.9397736 (231)\ttotal: 6.41s\tremaining: 18.9s\n",
            "253:\ttest: 0.9397604\tbest: 0.9397736 (231)\ttotal: 6.43s\tremaining: 18.9s\n",
            "254:\ttest: 0.9397703\tbest: 0.9397736 (231)\ttotal: 6.46s\tremaining: 18.9s\n",
            "255:\ttest: 0.9397839\tbest: 0.9397839 (255)\ttotal: 6.48s\tremaining: 18.8s\n",
            "256:\ttest: 0.9397900\tbest: 0.9397900 (256)\ttotal: 6.51s\tremaining: 18.8s\n",
            "257:\ttest: 0.9397629\tbest: 0.9397900 (256)\ttotal: 6.53s\tremaining: 18.8s\n",
            "258:\ttest: 0.9397905\tbest: 0.9397905 (258)\ttotal: 6.57s\tremaining: 18.8s\n",
            "259:\ttest: 0.9397880\tbest: 0.9397905 (258)\ttotal: 6.59s\tremaining: 18.8s\n",
            "260:\ttest: 0.9397832\tbest: 0.9397905 (258)\ttotal: 6.62s\tremaining: 18.7s\n",
            "261:\ttest: 0.9398030\tbest: 0.9398030 (261)\ttotal: 6.64s\tremaining: 18.7s\n",
            "262:\ttest: 0.9397867\tbest: 0.9398030 (261)\ttotal: 6.67s\tremaining: 18.7s\n",
            "263:\ttest: 0.9397929\tbest: 0.9398030 (261)\ttotal: 6.69s\tremaining: 18.7s\n",
            "264:\ttest: 0.9397995\tbest: 0.9398030 (261)\ttotal: 6.71s\tremaining: 18.6s\n",
            "265:\ttest: 0.9397992\tbest: 0.9398030 (261)\ttotal: 6.74s\tremaining: 18.6s\n",
            "266:\ttest: 0.9397999\tbest: 0.9398030 (261)\ttotal: 6.78s\tremaining: 18.6s\n",
            "267:\ttest: 0.9398033\tbest: 0.9398033 (267)\ttotal: 6.8s\tremaining: 18.6s\n",
            "268:\ttest: 0.9398212\tbest: 0.9398212 (268)\ttotal: 6.83s\tremaining: 18.6s\n",
            "269:\ttest: 0.9398365\tbest: 0.9398365 (269)\ttotal: 6.86s\tremaining: 18.5s\n",
            "270:\ttest: 0.9398561\tbest: 0.9398561 (270)\ttotal: 6.88s\tremaining: 18.5s\n",
            "271:\ttest: 0.9398669\tbest: 0.9398669 (271)\ttotal: 6.91s\tremaining: 18.5s\n",
            "272:\ttest: 0.9398826\tbest: 0.9398826 (272)\ttotal: 6.93s\tremaining: 18.5s\n",
            "273:\ttest: 0.9398967\tbest: 0.9398967 (273)\ttotal: 6.96s\tremaining: 18.4s\n",
            "274:\ttest: 0.9399076\tbest: 0.9399076 (274)\ttotal: 6.99s\tremaining: 18.4s\n",
            "275:\ttest: 0.9398998\tbest: 0.9399076 (274)\ttotal: 7.01s\tremaining: 18.4s\n",
            "276:\ttest: 0.9399054\tbest: 0.9399076 (274)\ttotal: 7.04s\tremaining: 18.4s\n",
            "277:\ttest: 0.9399149\tbest: 0.9399149 (277)\ttotal: 7.07s\tremaining: 18.4s\n",
            "278:\ttest: 0.9399242\tbest: 0.9399242 (278)\ttotal: 7.09s\tremaining: 18.3s\n",
            "279:\ttest: 0.9399320\tbest: 0.9399320 (279)\ttotal: 7.12s\tremaining: 18.3s\n",
            "280:\ttest: 0.9399344\tbest: 0.9399344 (280)\ttotal: 7.14s\tremaining: 18.3s\n",
            "281:\ttest: 0.9399300\tbest: 0.9399344 (280)\ttotal: 7.17s\tremaining: 18.3s\n",
            "282:\ttest: 0.9399320\tbest: 0.9399344 (280)\ttotal: 7.2s\tremaining: 18.2s\n",
            "283:\ttest: 0.9399450\tbest: 0.9399450 (283)\ttotal: 7.23s\tremaining: 18.2s\n",
            "284:\ttest: 0.9399440\tbest: 0.9399450 (283)\ttotal: 7.25s\tremaining: 18.2s\n",
            "285:\ttest: 0.9399460\tbest: 0.9399460 (285)\ttotal: 7.28s\tremaining: 18.2s\n",
            "286:\ttest: 0.9399415\tbest: 0.9399460 (285)\ttotal: 7.3s\tremaining: 18.1s\n",
            "287:\ttest: 0.9399357\tbest: 0.9399460 (285)\ttotal: 7.33s\tremaining: 18.1s\n",
            "288:\ttest: 0.9399459\tbest: 0.9399460 (285)\ttotal: 7.36s\tremaining: 18.1s\n",
            "289:\ttest: 0.9399454\tbest: 0.9399460 (285)\ttotal: 7.38s\tremaining: 18.1s\n",
            "290:\ttest: 0.9399478\tbest: 0.9399478 (290)\ttotal: 7.41s\tremaining: 18.1s\n",
            "291:\ttest: 0.9399774\tbest: 0.9399774 (291)\ttotal: 7.44s\tremaining: 18s\n",
            "292:\ttest: 0.9399883\tbest: 0.9399883 (292)\ttotal: 7.46s\tremaining: 18s\n",
            "293:\ttest: 0.9399908\tbest: 0.9399908 (293)\ttotal: 7.49s\tremaining: 18s\n",
            "294:\ttest: 0.9400009\tbest: 0.9400009 (294)\ttotal: 7.51s\tremaining: 18s\n",
            "295:\ttest: 0.9399982\tbest: 0.9400009 (294)\ttotal: 7.54s\tremaining: 17.9s\n",
            "296:\ttest: 0.9399918\tbest: 0.9400009 (294)\ttotal: 7.56s\tremaining: 17.9s\n",
            "297:\ttest: 0.9399943\tbest: 0.9400009 (294)\ttotal: 7.59s\tremaining: 17.9s\n",
            "298:\ttest: 0.9399985\tbest: 0.9400009 (294)\ttotal: 7.61s\tremaining: 17.8s\n",
            "299:\ttest: 0.9399959\tbest: 0.9400009 (294)\ttotal: 7.64s\tremaining: 17.8s\n",
            "300:\ttest: 0.9399972\tbest: 0.9400009 (294)\ttotal: 7.66s\tremaining: 17.8s\n",
            "301:\ttest: 0.9399888\tbest: 0.9400009 (294)\ttotal: 7.69s\tremaining: 17.8s\n",
            "302:\ttest: 0.9399969\tbest: 0.9400009 (294)\ttotal: 7.72s\tremaining: 17.8s\n",
            "303:\ttest: 0.9399948\tbest: 0.9400009 (294)\ttotal: 7.74s\tremaining: 17.7s\n",
            "304:\ttest: 0.9399779\tbest: 0.9400009 (294)\ttotal: 7.77s\tremaining: 17.7s\n",
            "305:\ttest: 0.9399743\tbest: 0.9400009 (294)\ttotal: 7.8s\tremaining: 17.7s\n",
            "306:\ttest: 0.9399853\tbest: 0.9400009 (294)\ttotal: 7.83s\tremaining: 17.7s\n",
            "307:\ttest: 0.9399742\tbest: 0.9400009 (294)\ttotal: 7.85s\tremaining: 17.6s\n",
            "308:\ttest: 0.9399731\tbest: 0.9400009 (294)\ttotal: 7.88s\tremaining: 17.6s\n",
            "309:\ttest: 0.9399744\tbest: 0.9400009 (294)\ttotal: 7.9s\tremaining: 17.6s\n",
            "310:\ttest: 0.9399812\tbest: 0.9400009 (294)\ttotal: 7.93s\tremaining: 17.6s\n",
            "311:\ttest: 0.9399880\tbest: 0.9400009 (294)\ttotal: 7.95s\tremaining: 17.5s\n",
            "312:\ttest: 0.9399973\tbest: 0.9400009 (294)\ttotal: 7.98s\tremaining: 17.5s\n",
            "313:\ttest: 0.9400081\tbest: 0.9400081 (313)\ttotal: 8s\tremaining: 17.5s\n",
            "314:\ttest: 0.9400113\tbest: 0.9400113 (314)\ttotal: 8.04s\tremaining: 17.5s\n",
            "315:\ttest: 0.9400011\tbest: 0.9400113 (314)\ttotal: 8.06s\tremaining: 17.4s\n",
            "316:\ttest: 0.9399969\tbest: 0.9400113 (314)\ttotal: 8.08s\tremaining: 17.4s\n",
            "317:\ttest: 0.9399789\tbest: 0.9400113 (314)\ttotal: 8.11s\tremaining: 17.4s\n",
            "318:\ttest: 0.9399954\tbest: 0.9400113 (314)\ttotal: 8.14s\tremaining: 17.4s\n",
            "319:\ttest: 0.9400102\tbest: 0.9400113 (314)\ttotal: 8.16s\tremaining: 17.3s\n",
            "320:\ttest: 0.9400106\tbest: 0.9400113 (314)\ttotal: 8.19s\tremaining: 17.3s\n",
            "321:\ttest: 0.9400041\tbest: 0.9400113 (314)\ttotal: 8.21s\tremaining: 17.3s\n",
            "322:\ttest: 0.9400096\tbest: 0.9400113 (314)\ttotal: 8.24s\tremaining: 17.3s\n",
            "323:\ttest: 0.9400074\tbest: 0.9400113 (314)\ttotal: 8.27s\tremaining: 17.3s\n",
            "324:\ttest: 0.9400114\tbest: 0.9400114 (324)\ttotal: 8.29s\tremaining: 17.2s\n",
            "325:\ttest: 0.9400128\tbest: 0.9400128 (325)\ttotal: 8.32s\tremaining: 17.2s\n",
            "326:\ttest: 0.9400142\tbest: 0.9400142 (326)\ttotal: 8.35s\tremaining: 17.2s\n",
            "327:\ttest: 0.9400332\tbest: 0.9400332 (327)\ttotal: 8.37s\tremaining: 17.2s\n",
            "328:\ttest: 0.9400415\tbest: 0.9400415 (328)\ttotal: 8.4s\tremaining: 17.1s\n",
            "329:\ttest: 0.9400490\tbest: 0.9400490 (329)\ttotal: 8.42s\tremaining: 17.1s\n",
            "330:\ttest: 0.9400378\tbest: 0.9400490 (329)\ttotal: 8.46s\tremaining: 17.1s\n",
            "331:\ttest: 0.9400373\tbest: 0.9400490 (329)\ttotal: 8.48s\tremaining: 17.1s\n",
            "332:\ttest: 0.9400427\tbest: 0.9400490 (329)\ttotal: 8.51s\tremaining: 17s\n",
            "333:\ttest: 0.9400383\tbest: 0.9400490 (329)\ttotal: 8.53s\tremaining: 17s\n",
            "334:\ttest: 0.9400352\tbest: 0.9400490 (329)\ttotal: 8.56s\tremaining: 17s\n",
            "335:\ttest: 0.9400474\tbest: 0.9400490 (329)\ttotal: 8.58s\tremaining: 17s\n",
            "336:\ttest: 0.9400452\tbest: 0.9400490 (329)\ttotal: 8.61s\tremaining: 16.9s\n",
            "337:\ttest: 0.9400503\tbest: 0.9400503 (337)\ttotal: 8.63s\tremaining: 16.9s\n",
            "338:\ttest: 0.9400587\tbest: 0.9400587 (338)\ttotal: 8.66s\tremaining: 16.9s\n",
            "339:\ttest: 0.9400625\tbest: 0.9400625 (339)\ttotal: 8.69s\tremaining: 16.9s\n",
            "340:\ttest: 0.9400431\tbest: 0.9400625 (339)\ttotal: 8.72s\tremaining: 16.9s\n",
            "341:\ttest: 0.9400683\tbest: 0.9400683 (341)\ttotal: 8.75s\tremaining: 16.8s\n",
            "342:\ttest: 0.9400734\tbest: 0.9400734 (342)\ttotal: 8.77s\tremaining: 16.8s\n",
            "343:\ttest: 0.9400905\tbest: 0.9400905 (343)\ttotal: 8.79s\tremaining: 16.8s\n",
            "344:\ttest: 0.9401173\tbest: 0.9401173 (344)\ttotal: 8.82s\tremaining: 16.7s\n",
            "345:\ttest: 0.9401208\tbest: 0.9401208 (345)\ttotal: 8.85s\tremaining: 16.7s\n",
            "346:\ttest: 0.9401387\tbest: 0.9401387 (346)\ttotal: 8.88s\tremaining: 16.7s\n",
            "347:\ttest: 0.9401342\tbest: 0.9401387 (346)\ttotal: 8.9s\tremaining: 16.7s\n",
            "348:\ttest: 0.9401426\tbest: 0.9401426 (348)\ttotal: 8.93s\tremaining: 16.7s\n",
            "349:\ttest: 0.9401553\tbest: 0.9401553 (349)\ttotal: 8.95s\tremaining: 16.6s\n",
            "350:\ttest: 0.9401592\tbest: 0.9401592 (350)\ttotal: 8.98s\tremaining: 16.6s\n",
            "351:\ttest: 0.9401681\tbest: 0.9401681 (351)\ttotal: 9.01s\tremaining: 16.6s\n",
            "352:\ttest: 0.9401431\tbest: 0.9401681 (351)\ttotal: 9.04s\tremaining: 16.6s\n",
            "353:\ttest: 0.9401484\tbest: 0.9401681 (351)\ttotal: 9.06s\tremaining: 16.5s\n",
            "354:\ttest: 0.9401629\tbest: 0.9401681 (351)\ttotal: 9.09s\tremaining: 16.5s\n",
            "355:\ttest: 0.9401670\tbest: 0.9401681 (351)\ttotal: 9.12s\tremaining: 16.5s\n",
            "356:\ttest: 0.9401644\tbest: 0.9401681 (351)\ttotal: 9.14s\tremaining: 16.5s\n",
            "357:\ttest: 0.9401578\tbest: 0.9401681 (351)\ttotal: 9.17s\tremaining: 16.4s\n",
            "358:\ttest: 0.9401600\tbest: 0.9401681 (351)\ttotal: 9.19s\tremaining: 16.4s\n",
            "359:\ttest: 0.9401664\tbest: 0.9401681 (351)\ttotal: 9.22s\tremaining: 16.4s\n",
            "360:\ttest: 0.9401950\tbest: 0.9401950 (360)\ttotal: 9.24s\tremaining: 16.4s\n",
            "361:\ttest: 0.9402097\tbest: 0.9402097 (361)\ttotal: 9.27s\tremaining: 16.3s\n",
            "362:\ttest: 0.9402200\tbest: 0.9402200 (362)\ttotal: 9.29s\tremaining: 16.3s\n",
            "363:\ttest: 0.9402260\tbest: 0.9402260 (363)\ttotal: 9.33s\tremaining: 16.3s\n",
            "364:\ttest: 0.9402301\tbest: 0.9402301 (364)\ttotal: 9.35s\tremaining: 16.3s\n",
            "365:\ttest: 0.9402257\tbest: 0.9402301 (364)\ttotal: 9.38s\tremaining: 16.2s\n",
            "366:\ttest: 0.9402179\tbest: 0.9402301 (364)\ttotal: 9.4s\tremaining: 16.2s\n",
            "367:\ttest: 0.9402222\tbest: 0.9402301 (364)\ttotal: 9.43s\tremaining: 16.2s\n",
            "368:\ttest: 0.9402462\tbest: 0.9402462 (368)\ttotal: 9.45s\tremaining: 16.2s\n",
            "369:\ttest: 0.9402475\tbest: 0.9402475 (369)\ttotal: 9.48s\tremaining: 16.1s\n",
            "370:\ttest: 0.9402428\tbest: 0.9402475 (369)\ttotal: 9.51s\tremaining: 16.1s\n",
            "371:\ttest: 0.9402605\tbest: 0.9402605 (371)\ttotal: 9.53s\tremaining: 16.1s\n",
            "372:\ttest: 0.9402807\tbest: 0.9402807 (372)\ttotal: 9.56s\tremaining: 16.1s\n",
            "373:\ttest: 0.9402907\tbest: 0.9402907 (373)\ttotal: 9.58s\tremaining: 16s\n",
            "374:\ttest: 0.9402995\tbest: 0.9402995 (374)\ttotal: 9.61s\tremaining: 16s\n",
            "375:\ttest: 0.9402871\tbest: 0.9402995 (374)\ttotal: 9.63s\tremaining: 16s\n",
            "376:\ttest: 0.9402947\tbest: 0.9402995 (374)\ttotal: 9.66s\tremaining: 16s\n",
            "377:\ttest: 0.9402900\tbest: 0.9402995 (374)\ttotal: 9.69s\tremaining: 15.9s\n",
            "378:\ttest: 0.9403114\tbest: 0.9403114 (378)\ttotal: 9.72s\tremaining: 15.9s\n",
            "379:\ttest: 0.9403005\tbest: 0.9403114 (378)\ttotal: 9.75s\tremaining: 15.9s\n",
            "380:\ttest: 0.9403184\tbest: 0.9403184 (380)\ttotal: 9.77s\tremaining: 15.9s\n",
            "381:\ttest: 0.9403210\tbest: 0.9403210 (381)\ttotal: 9.8s\tremaining: 15.9s\n",
            "382:\ttest: 0.9403199\tbest: 0.9403210 (381)\ttotal: 9.82s\tremaining: 15.8s\n",
            "383:\ttest: 0.9403245\tbest: 0.9403245 (383)\ttotal: 9.85s\tremaining: 15.8s\n",
            "384:\ttest: 0.9403301\tbest: 0.9403301 (384)\ttotal: 9.87s\tremaining: 15.8s\n",
            "385:\ttest: 0.9403265\tbest: 0.9403301 (384)\ttotal: 9.9s\tremaining: 15.7s\n",
            "386:\ttest: 0.9403314\tbest: 0.9403314 (386)\ttotal: 9.93s\tremaining: 15.7s\n",
            "387:\ttest: 0.9403364\tbest: 0.9403364 (387)\ttotal: 9.95s\tremaining: 15.7s\n",
            "388:\ttest: 0.9403329\tbest: 0.9403364 (387)\ttotal: 9.98s\tremaining: 15.7s\n",
            "389:\ttest: 0.9403396\tbest: 0.9403396 (389)\ttotal: 10s\tremaining: 15.7s\n",
            "390:\ttest: 0.9403522\tbest: 0.9403522 (390)\ttotal: 10s\tremaining: 15.6s\n",
            "391:\ttest: 0.9403467\tbest: 0.9403522 (390)\ttotal: 10.1s\tremaining: 15.6s\n",
            "392:\ttest: 0.9403592\tbest: 0.9403592 (392)\ttotal: 10.1s\tremaining: 15.6s\n",
            "393:\ttest: 0.9403529\tbest: 0.9403592 (392)\ttotal: 10.1s\tremaining: 15.5s\n",
            "394:\ttest: 0.9403609\tbest: 0.9403609 (394)\ttotal: 10.1s\tremaining: 15.5s\n",
            "395:\ttest: 0.9403693\tbest: 0.9403693 (395)\ttotal: 10.2s\tremaining: 15.5s\n",
            "396:\ttest: 0.9403721\tbest: 0.9403721 (396)\ttotal: 10.2s\tremaining: 15.5s\n",
            "397:\ttest: 0.9403775\tbest: 0.9403775 (397)\ttotal: 10.2s\tremaining: 15.5s\n",
            "398:\ttest: 0.9403893\tbest: 0.9403893 (398)\ttotal: 10.2s\tremaining: 15.4s\n",
            "399:\ttest: 0.9403847\tbest: 0.9403893 (398)\ttotal: 10.3s\tremaining: 15.4s\n",
            "400:\ttest: 0.9403891\tbest: 0.9403893 (398)\ttotal: 10.3s\tremaining: 15.4s\n",
            "401:\ttest: 0.9403825\tbest: 0.9403893 (398)\ttotal: 10.3s\tremaining: 15.3s\n",
            "402:\ttest: 0.9403772\tbest: 0.9403893 (398)\ttotal: 10.4s\tremaining: 15.3s\n",
            "403:\ttest: 0.9403740\tbest: 0.9403893 (398)\ttotal: 10.4s\tremaining: 15.3s\n",
            "404:\ttest: 0.9403760\tbest: 0.9403893 (398)\ttotal: 10.4s\tremaining: 15.3s\n",
            "405:\ttest: 0.9403881\tbest: 0.9403893 (398)\ttotal: 10.4s\tremaining: 15.3s\n",
            "406:\ttest: 0.9403955\tbest: 0.9403955 (406)\ttotal: 10.5s\tremaining: 15.2s\n",
            "407:\ttest: 0.9403981\tbest: 0.9403981 (407)\ttotal: 10.5s\tremaining: 15.2s\n",
            "408:\ttest: 0.9403994\tbest: 0.9403994 (408)\ttotal: 10.5s\tremaining: 15.2s\n",
            "409:\ttest: 0.9403985\tbest: 0.9403994 (408)\ttotal: 10.5s\tremaining: 15.2s\n",
            "410:\ttest: 0.9404162\tbest: 0.9404162 (410)\ttotal: 10.6s\tremaining: 15.1s\n",
            "411:\ttest: 0.9404118\tbest: 0.9404162 (410)\ttotal: 10.6s\tremaining: 15.1s\n",
            "412:\ttest: 0.9404163\tbest: 0.9404163 (412)\ttotal: 10.6s\tremaining: 15.1s\n",
            "413:\ttest: 0.9404164\tbest: 0.9404164 (413)\ttotal: 10.6s\tremaining: 15s\n",
            "414:\ttest: 0.9404175\tbest: 0.9404175 (414)\ttotal: 10.7s\tremaining: 15s\n",
            "415:\ttest: 0.9404090\tbest: 0.9404175 (414)\ttotal: 10.7s\tremaining: 15s\n",
            "416:\ttest: 0.9404217\tbest: 0.9404217 (416)\ttotal: 10.7s\tremaining: 15s\n",
            "417:\ttest: 0.9404302\tbest: 0.9404302 (417)\ttotal: 10.7s\tremaining: 14.9s\n",
            "418:\ttest: 0.9404337\tbest: 0.9404337 (418)\ttotal: 10.8s\tremaining: 14.9s\n",
            "419:\ttest: 0.9404187\tbest: 0.9404337 (418)\ttotal: 10.8s\tremaining: 14.9s\n",
            "420:\ttest: 0.9404245\tbest: 0.9404337 (418)\ttotal: 10.8s\tremaining: 14.9s\n",
            "421:\ttest: 0.9404190\tbest: 0.9404337 (418)\ttotal: 10.8s\tremaining: 14.9s\n",
            "422:\ttest: 0.9404236\tbest: 0.9404337 (418)\ttotal: 10.9s\tremaining: 14.8s\n",
            "423:\ttest: 0.9404092\tbest: 0.9404337 (418)\ttotal: 10.9s\tremaining: 14.8s\n",
            "424:\ttest: 0.9404181\tbest: 0.9404337 (418)\ttotal: 10.9s\tremaining: 14.8s\n",
            "425:\ttest: 0.9404112\tbest: 0.9404337 (418)\ttotal: 11s\tremaining: 14.8s\n",
            "426:\ttest: 0.9404064\tbest: 0.9404337 (418)\ttotal: 11s\tremaining: 14.7s\n",
            "427:\ttest: 0.9403980\tbest: 0.9404337 (418)\ttotal: 11s\tremaining: 14.7s\n",
            "428:\ttest: 0.9404016\tbest: 0.9404337 (418)\ttotal: 11s\tremaining: 14.7s\n",
            "429:\ttest: 0.9404072\tbest: 0.9404337 (418)\ttotal: 11.1s\tremaining: 14.7s\n",
            "430:\ttest: 0.9403935\tbest: 0.9404337 (418)\ttotal: 11.1s\tremaining: 14.6s\n",
            "431:\ttest: 0.9404114\tbest: 0.9404337 (418)\ttotal: 11.1s\tremaining: 14.6s\n",
            "432:\ttest: 0.9404101\tbest: 0.9404337 (418)\ttotal: 11.1s\tremaining: 14.6s\n",
            "433:\ttest: 0.9404211\tbest: 0.9404337 (418)\ttotal: 11.2s\tremaining: 14.6s\n",
            "434:\ttest: 0.9404200\tbest: 0.9404337 (418)\ttotal: 11.2s\tremaining: 14.5s\n",
            "435:\ttest: 0.9404102\tbest: 0.9404337 (418)\ttotal: 11.2s\tremaining: 14.5s\n",
            "436:\ttest: 0.9404156\tbest: 0.9404337 (418)\ttotal: 11.2s\tremaining: 14.5s\n",
            "437:\ttest: 0.9404297\tbest: 0.9404337 (418)\ttotal: 11.3s\tremaining: 14.5s\n",
            "438:\ttest: 0.9404208\tbest: 0.9404337 (418)\ttotal: 11.3s\tremaining: 14.4s\n",
            "439:\ttest: 0.9404255\tbest: 0.9404337 (418)\ttotal: 11.3s\tremaining: 14.4s\n",
            "440:\ttest: 0.9404250\tbest: 0.9404337 (418)\ttotal: 11.3s\tremaining: 14.4s\n",
            "441:\ttest: 0.9404260\tbest: 0.9404337 (418)\ttotal: 11.4s\tremaining: 14.3s\n",
            "442:\ttest: 0.9404181\tbest: 0.9404337 (418)\ttotal: 11.4s\tremaining: 14.3s\n",
            "443:\ttest: 0.9404300\tbest: 0.9404337 (418)\ttotal: 11.4s\tremaining: 14.3s\n",
            "444:\ttest: 0.9404296\tbest: 0.9404337 (418)\ttotal: 11.4s\tremaining: 14.3s\n",
            "445:\ttest: 0.9404590\tbest: 0.9404590 (445)\ttotal: 11.5s\tremaining: 14.2s\n",
            "446:\ttest: 0.9404510\tbest: 0.9404590 (445)\ttotal: 11.5s\tremaining: 14.2s\n",
            "447:\ttest: 0.9404580\tbest: 0.9404590 (445)\ttotal: 11.5s\tremaining: 14.2s\n",
            "448:\ttest: 0.9404553\tbest: 0.9404590 (445)\ttotal: 11.5s\tremaining: 14.2s\n",
            "449:\ttest: 0.9404682\tbest: 0.9404682 (449)\ttotal: 11.6s\tremaining: 14.1s\n",
            "450:\ttest: 0.9404740\tbest: 0.9404740 (450)\ttotal: 11.6s\tremaining: 14.1s\n",
            "451:\ttest: 0.9404709\tbest: 0.9404740 (450)\ttotal: 11.6s\tremaining: 14.1s\n",
            "452:\ttest: 0.9404802\tbest: 0.9404802 (452)\ttotal: 11.6s\tremaining: 14s\n",
            "453:\ttest: 0.9404800\tbest: 0.9404802 (452)\ttotal: 11.7s\tremaining: 14s\n",
            "454:\ttest: 0.9404828\tbest: 0.9404828 (454)\ttotal: 11.7s\tremaining: 14s\n",
            "455:\ttest: 0.9404888\tbest: 0.9404888 (455)\ttotal: 11.7s\tremaining: 14s\n",
            "456:\ttest: 0.9404891\tbest: 0.9404891 (456)\ttotal: 11.7s\tremaining: 13.9s\n",
            "457:\ttest: 0.9404843\tbest: 0.9404891 (456)\ttotal: 11.8s\tremaining: 13.9s\n",
            "458:\ttest: 0.9404881\tbest: 0.9404891 (456)\ttotal: 11.8s\tremaining: 13.9s\n",
            "459:\ttest: 0.9404894\tbest: 0.9404894 (459)\ttotal: 11.8s\tremaining: 13.9s\n",
            "460:\ttest: 0.9404833\tbest: 0.9404894 (459)\ttotal: 11.8s\tremaining: 13.8s\n",
            "461:\ttest: 0.9404913\tbest: 0.9404913 (461)\ttotal: 11.9s\tremaining: 13.8s\n",
            "462:\ttest: 0.9404971\tbest: 0.9404971 (462)\ttotal: 11.9s\tremaining: 13.8s\n",
            "463:\ttest: 0.9405106\tbest: 0.9405106 (463)\ttotal: 11.9s\tremaining: 13.8s\n",
            "464:\ttest: 0.9405156\tbest: 0.9405156 (464)\ttotal: 12s\tremaining: 13.8s\n",
            "465:\ttest: 0.9405155\tbest: 0.9405156 (464)\ttotal: 12s\tremaining: 13.8s\n",
            "466:\ttest: 0.9405330\tbest: 0.9405330 (466)\ttotal: 12s\tremaining: 13.7s\n",
            "467:\ttest: 0.9405314\tbest: 0.9405330 (466)\ttotal: 12.1s\tremaining: 13.7s\n",
            "468:\ttest: 0.9405263\tbest: 0.9405330 (466)\ttotal: 12.1s\tremaining: 13.7s\n",
            "469:\ttest: 0.9405251\tbest: 0.9405330 (466)\ttotal: 12.1s\tremaining: 13.7s\n",
            "470:\ttest: 0.9405305\tbest: 0.9405330 (466)\ttotal: 12.1s\tremaining: 13.6s\n",
            "471:\ttest: 0.9405292\tbest: 0.9405330 (466)\ttotal: 12.2s\tremaining: 13.6s\n",
            "472:\ttest: 0.9405360\tbest: 0.9405360 (472)\ttotal: 12.2s\tremaining: 13.6s\n",
            "473:\ttest: 0.9405422\tbest: 0.9405422 (473)\ttotal: 12.2s\tremaining: 13.5s\n",
            "474:\ttest: 0.9405549\tbest: 0.9405549 (474)\ttotal: 12.2s\tremaining: 13.5s\n",
            "475:\ttest: 0.9405621\tbest: 0.9405621 (475)\ttotal: 12.3s\tremaining: 13.5s\n",
            "476:\ttest: 0.9405738\tbest: 0.9405738 (476)\ttotal: 12.3s\tremaining: 13.5s\n",
            "477:\ttest: 0.9405818\tbest: 0.9405818 (477)\ttotal: 12.3s\tremaining: 13.4s\n",
            "478:\ttest: 0.9405893\tbest: 0.9405893 (478)\ttotal: 12.3s\tremaining: 13.4s\n",
            "479:\ttest: 0.9405892\tbest: 0.9405893 (478)\ttotal: 12.4s\tremaining: 13.4s\n",
            "480:\ttest: 0.9406023\tbest: 0.9406023 (480)\ttotal: 12.4s\tremaining: 13.4s\n",
            "481:\ttest: 0.9406032\tbest: 0.9406032 (481)\ttotal: 12.4s\tremaining: 13.3s\n",
            "482:\ttest: 0.9405982\tbest: 0.9406032 (481)\ttotal: 12.4s\tremaining: 13.3s\n",
            "483:\ttest: 0.9406067\tbest: 0.9406067 (483)\ttotal: 12.5s\tremaining: 13.3s\n",
            "484:\ttest: 0.9406118\tbest: 0.9406118 (484)\ttotal: 12.5s\tremaining: 13.3s\n",
            "485:\ttest: 0.9406137\tbest: 0.9406137 (485)\ttotal: 12.5s\tremaining: 13.2s\n",
            "486:\ttest: 0.9406124\tbest: 0.9406137 (485)\ttotal: 12.5s\tremaining: 13.2s\n",
            "487:\ttest: 0.9406026\tbest: 0.9406137 (485)\ttotal: 12.6s\tremaining: 13.2s\n",
            "488:\ttest: 0.9406074\tbest: 0.9406137 (485)\ttotal: 12.6s\tremaining: 13.2s\n",
            "489:\ttest: 0.9406027\tbest: 0.9406137 (485)\ttotal: 12.6s\tremaining: 13.1s\n",
            "490:\ttest: 0.9406253\tbest: 0.9406253 (490)\ttotal: 12.7s\tremaining: 13.1s\n",
            "491:\ttest: 0.9406315\tbest: 0.9406315 (491)\ttotal: 12.7s\tremaining: 13.1s\n",
            "492:\ttest: 0.9406329\tbest: 0.9406329 (492)\ttotal: 12.7s\tremaining: 13.1s\n",
            "493:\ttest: 0.9406288\tbest: 0.9406329 (492)\ttotal: 12.7s\tremaining: 13s\n",
            "494:\ttest: 0.9406209\tbest: 0.9406329 (492)\ttotal: 12.8s\tremaining: 13s\n",
            "495:\ttest: 0.9406265\tbest: 0.9406329 (492)\ttotal: 12.8s\tremaining: 13s\n",
            "496:\ttest: 0.9406161\tbest: 0.9406329 (492)\ttotal: 12.8s\tremaining: 13s\n",
            "497:\ttest: 0.9406176\tbest: 0.9406329 (492)\ttotal: 12.8s\tremaining: 12.9s\n",
            "498:\ttest: 0.9406180\tbest: 0.9406329 (492)\ttotal: 12.9s\tremaining: 12.9s\n",
            "499:\ttest: 0.9406210\tbest: 0.9406329 (492)\ttotal: 12.9s\tremaining: 12.9s\n",
            "500:\ttest: 0.9406129\tbest: 0.9406329 (492)\ttotal: 12.9s\tremaining: 12.9s\n",
            "501:\ttest: 0.9406145\tbest: 0.9406329 (492)\ttotal: 12.9s\tremaining: 12.8s\n",
            "502:\ttest: 0.9406180\tbest: 0.9406329 (492)\ttotal: 13s\tremaining: 12.8s\n",
            "503:\ttest: 0.9406182\tbest: 0.9406329 (492)\ttotal: 13s\tremaining: 12.8s\n",
            "504:\ttest: 0.9406195\tbest: 0.9406329 (492)\ttotal: 13s\tremaining: 12.7s\n",
            "505:\ttest: 0.9406096\tbest: 0.9406329 (492)\ttotal: 13s\tremaining: 12.7s\n",
            "506:\ttest: 0.9406130\tbest: 0.9406329 (492)\ttotal: 13.1s\tremaining: 12.7s\n",
            "507:\ttest: 0.9406137\tbest: 0.9406329 (492)\ttotal: 13.1s\tremaining: 12.7s\n",
            "508:\ttest: 0.9406240\tbest: 0.9406329 (492)\ttotal: 13.1s\tremaining: 12.7s\n",
            "509:\ttest: 0.9406206\tbest: 0.9406329 (492)\ttotal: 13.1s\tremaining: 12.6s\n",
            "510:\ttest: 0.9406239\tbest: 0.9406329 (492)\ttotal: 13.2s\tremaining: 12.6s\n",
            "511:\ttest: 0.9406219\tbest: 0.9406329 (492)\ttotal: 13.2s\tremaining: 12.6s\n",
            "512:\ttest: 0.9406210\tbest: 0.9406329 (492)\ttotal: 13.2s\tremaining: 12.6s\n",
            "513:\ttest: 0.9406226\tbest: 0.9406329 (492)\ttotal: 13.2s\tremaining: 12.5s\n",
            "514:\ttest: 0.9406327\tbest: 0.9406329 (492)\ttotal: 13.3s\tremaining: 12.5s\n",
            "515:\ttest: 0.9406327\tbest: 0.9406329 (492)\ttotal: 13.3s\tremaining: 12.5s\n",
            "516:\ttest: 0.9406328\tbest: 0.9406329 (492)\ttotal: 13.3s\tremaining: 12.5s\n",
            "517:\ttest: 0.9406364\tbest: 0.9406364 (517)\ttotal: 13.4s\tremaining: 12.4s\n",
            "518:\ttest: 0.9406433\tbest: 0.9406433 (518)\ttotal: 13.4s\tremaining: 12.4s\n",
            "519:\ttest: 0.9406492\tbest: 0.9406492 (519)\ttotal: 13.4s\tremaining: 12.4s\n",
            "520:\ttest: 0.9406455\tbest: 0.9406492 (519)\ttotal: 13.4s\tremaining: 12.3s\n",
            "521:\ttest: 0.9406544\tbest: 0.9406544 (521)\ttotal: 13.5s\tremaining: 12.3s\n",
            "522:\ttest: 0.9406625\tbest: 0.9406625 (522)\ttotal: 13.5s\tremaining: 12.3s\n",
            "523:\ttest: 0.9406739\tbest: 0.9406739 (523)\ttotal: 13.5s\tremaining: 12.3s\n",
            "524:\ttest: 0.9406682\tbest: 0.9406739 (523)\ttotal: 13.5s\tremaining: 12.2s\n",
            "525:\ttest: 0.9406794\tbest: 0.9406794 (525)\ttotal: 13.6s\tremaining: 12.2s\n",
            "526:\ttest: 0.9406913\tbest: 0.9406913 (526)\ttotal: 13.6s\tremaining: 12.2s\n",
            "527:\ttest: 0.9406946\tbest: 0.9406946 (527)\ttotal: 13.6s\tremaining: 12.2s\n",
            "528:\ttest: 0.9406913\tbest: 0.9406946 (527)\ttotal: 13.6s\tremaining: 12.1s\n",
            "529:\ttest: 0.9406838\tbest: 0.9406946 (527)\ttotal: 13.7s\tremaining: 12.1s\n",
            "530:\ttest: 0.9406852\tbest: 0.9406946 (527)\ttotal: 13.7s\tremaining: 12.1s\n",
            "531:\ttest: 0.9406834\tbest: 0.9406946 (527)\ttotal: 13.7s\tremaining: 12.1s\n",
            "532:\ttest: 0.9406912\tbest: 0.9406946 (527)\ttotal: 13.7s\tremaining: 12s\n",
            "533:\ttest: 0.9406895\tbest: 0.9406946 (527)\ttotal: 13.8s\tremaining: 12s\n",
            "534:\ttest: 0.9406877\tbest: 0.9406946 (527)\ttotal: 13.8s\tremaining: 12s\n",
            "535:\ttest: 0.9406923\tbest: 0.9406946 (527)\ttotal: 13.8s\tremaining: 12s\n",
            "536:\ttest: 0.9407027\tbest: 0.9407027 (536)\ttotal: 13.8s\tremaining: 11.9s\n",
            "537:\ttest: 0.9407024\tbest: 0.9407027 (536)\ttotal: 13.9s\tremaining: 11.9s\n",
            "538:\ttest: 0.9406999\tbest: 0.9407027 (536)\ttotal: 13.9s\tremaining: 11.9s\n",
            "539:\ttest: 0.9407145\tbest: 0.9407145 (539)\ttotal: 13.9s\tremaining: 11.8s\n",
            "540:\ttest: 0.9407118\tbest: 0.9407145 (539)\ttotal: 13.9s\tremaining: 11.8s\n",
            "541:\ttest: 0.9407147\tbest: 0.9407147 (541)\ttotal: 14s\tremaining: 11.8s\n",
            "542:\ttest: 0.9407262\tbest: 0.9407262 (542)\ttotal: 14s\tremaining: 11.8s\n",
            "543:\ttest: 0.9407276\tbest: 0.9407276 (543)\ttotal: 14s\tremaining: 11.8s\n",
            "544:\ttest: 0.9407235\tbest: 0.9407276 (543)\ttotal: 14s\tremaining: 11.7s\n",
            "545:\ttest: 0.9407295\tbest: 0.9407295 (545)\ttotal: 14.1s\tremaining: 11.7s\n",
            "546:\ttest: 0.9407292\tbest: 0.9407295 (545)\ttotal: 14.1s\tremaining: 11.7s\n",
            "547:\ttest: 0.9407228\tbest: 0.9407295 (545)\ttotal: 14.1s\tremaining: 11.7s\n",
            "548:\ttest: 0.9407242\tbest: 0.9407295 (545)\ttotal: 14.2s\tremaining: 11.6s\n",
            "549:\ttest: 0.9407292\tbest: 0.9407295 (545)\ttotal: 14.2s\tremaining: 11.6s\n",
            "550:\ttest: 0.9407166\tbest: 0.9407295 (545)\ttotal: 14.2s\tremaining: 11.6s\n",
            "551:\ttest: 0.9407195\tbest: 0.9407295 (545)\ttotal: 14.2s\tremaining: 11.5s\n",
            "552:\ttest: 0.9407173\tbest: 0.9407295 (545)\ttotal: 14.3s\tremaining: 11.5s\n",
            "553:\ttest: 0.9407095\tbest: 0.9407295 (545)\ttotal: 14.3s\tremaining: 11.5s\n",
            "554:\ttest: 0.9407148\tbest: 0.9407295 (545)\ttotal: 14.3s\tremaining: 11.5s\n",
            "555:\ttest: 0.9407085\tbest: 0.9407295 (545)\ttotal: 14.3s\tremaining: 11.4s\n",
            "556:\ttest: 0.9407105\tbest: 0.9407295 (545)\ttotal: 14.4s\tremaining: 11.4s\n",
            "557:\ttest: 0.9407277\tbest: 0.9407295 (545)\ttotal: 14.4s\tremaining: 11.4s\n",
            "558:\ttest: 0.9407400\tbest: 0.9407400 (558)\ttotal: 14.4s\tremaining: 11.4s\n",
            "559:\ttest: 0.9407475\tbest: 0.9407475 (559)\ttotal: 14.4s\tremaining: 11.3s\n",
            "560:\ttest: 0.9407580\tbest: 0.9407580 (560)\ttotal: 14.5s\tremaining: 11.3s\n",
            "561:\ttest: 0.9407555\tbest: 0.9407580 (560)\ttotal: 14.5s\tremaining: 11.3s\n",
            "562:\ttest: 0.9407638\tbest: 0.9407638 (562)\ttotal: 14.5s\tremaining: 11.3s\n",
            "563:\ttest: 0.9407581\tbest: 0.9407638 (562)\ttotal: 14.5s\tremaining: 11.2s\n",
            "564:\ttest: 0.9407598\tbest: 0.9407638 (562)\ttotal: 14.6s\tremaining: 11.2s\n",
            "565:\ttest: 0.9407592\tbest: 0.9407638 (562)\ttotal: 14.6s\tremaining: 11.2s\n",
            "566:\ttest: 0.9407680\tbest: 0.9407680 (566)\ttotal: 14.6s\tremaining: 11.2s\n",
            "567:\ttest: 0.9407633\tbest: 0.9407680 (566)\ttotal: 14.6s\tremaining: 11.1s\n",
            "568:\ttest: 0.9407661\tbest: 0.9407680 (566)\ttotal: 14.7s\tremaining: 11.1s\n",
            "569:\ttest: 0.9407603\tbest: 0.9407680 (566)\ttotal: 14.7s\tremaining: 11.1s\n",
            "570:\ttest: 0.9407605\tbest: 0.9407680 (566)\ttotal: 14.7s\tremaining: 11.1s\n",
            "571:\ttest: 0.9407630\tbest: 0.9407680 (566)\ttotal: 14.7s\tremaining: 11s\n",
            "572:\ttest: 0.9407669\tbest: 0.9407680 (566)\ttotal: 14.8s\tremaining: 11s\n",
            "573:\ttest: 0.9407701\tbest: 0.9407701 (573)\ttotal: 14.8s\tremaining: 11s\n",
            "574:\ttest: 0.9407831\tbest: 0.9407831 (574)\ttotal: 14.8s\tremaining: 11s\n",
            "575:\ttest: 0.9407752\tbest: 0.9407831 (574)\ttotal: 14.9s\tremaining: 10.9s\n",
            "576:\ttest: 0.9407854\tbest: 0.9407854 (576)\ttotal: 14.9s\tremaining: 10.9s\n",
            "577:\ttest: 0.9407645\tbest: 0.9407854 (576)\ttotal: 14.9s\tremaining: 10.9s\n",
            "578:\ttest: 0.9407637\tbest: 0.9407854 (576)\ttotal: 14.9s\tremaining: 10.9s\n",
            "579:\ttest: 0.9407593\tbest: 0.9407854 (576)\ttotal: 15s\tremaining: 10.8s\n",
            "580:\ttest: 0.9407625\tbest: 0.9407854 (576)\ttotal: 15s\tremaining: 10.8s\n",
            "581:\ttest: 0.9407542\tbest: 0.9407854 (576)\ttotal: 15s\tremaining: 10.8s\n",
            "582:\ttest: 0.9407492\tbest: 0.9407854 (576)\ttotal: 15.1s\tremaining: 10.8s\n",
            "583:\ttest: 0.9407486\tbest: 0.9407854 (576)\ttotal: 15.1s\tremaining: 10.7s\n",
            "584:\ttest: 0.9407470\tbest: 0.9407854 (576)\ttotal: 15.1s\tremaining: 10.7s\n",
            "585:\ttest: 0.9407448\tbest: 0.9407854 (576)\ttotal: 15.1s\tremaining: 10.7s\n",
            "586:\ttest: 0.9407401\tbest: 0.9407854 (576)\ttotal: 15.2s\tremaining: 10.7s\n",
            "587:\ttest: 0.9407335\tbest: 0.9407854 (576)\ttotal: 15.2s\tremaining: 10.6s\n",
            "588:\ttest: 0.9407345\tbest: 0.9407854 (576)\ttotal: 15.2s\tremaining: 10.6s\n",
            "589:\ttest: 0.9407426\tbest: 0.9407854 (576)\ttotal: 15.2s\tremaining: 10.6s\n",
            "590:\ttest: 0.9407393\tbest: 0.9407854 (576)\ttotal: 15.3s\tremaining: 10.6s\n",
            "591:\ttest: 0.9407350\tbest: 0.9407854 (576)\ttotal: 15.3s\tremaining: 10.5s\n",
            "592:\ttest: 0.9407373\tbest: 0.9407854 (576)\ttotal: 15.3s\tremaining: 10.5s\n",
            "593:\ttest: 0.9407328\tbest: 0.9407854 (576)\ttotal: 15.3s\tremaining: 10.5s\n",
            "594:\ttest: 0.9407341\tbest: 0.9407854 (576)\ttotal: 15.4s\tremaining: 10.5s\n",
            "595:\ttest: 0.9407332\tbest: 0.9407854 (576)\ttotal: 15.4s\tremaining: 10.4s\n",
            "596:\ttest: 0.9407418\tbest: 0.9407854 (576)\ttotal: 15.4s\tremaining: 10.4s\n",
            "597:\ttest: 0.9407483\tbest: 0.9407854 (576)\ttotal: 15.4s\tremaining: 10.4s\n",
            "598:\ttest: 0.9407493\tbest: 0.9407854 (576)\ttotal: 15.5s\tremaining: 10.4s\n",
            "599:\ttest: 0.9407596\tbest: 0.9407854 (576)\ttotal: 15.5s\tremaining: 10.3s\n",
            "600:\ttest: 0.9407526\tbest: 0.9407854 (576)\ttotal: 15.5s\tremaining: 10.3s\n",
            "601:\ttest: 0.9407669\tbest: 0.9407854 (576)\ttotal: 15.5s\tremaining: 10.3s\n",
            "602:\ttest: 0.9407672\tbest: 0.9407854 (576)\ttotal: 15.6s\tremaining: 10.2s\n",
            "603:\ttest: 0.9407675\tbest: 0.9407854 (576)\ttotal: 15.6s\tremaining: 10.2s\n",
            "604:\ttest: 0.9407778\tbest: 0.9407854 (576)\ttotal: 15.6s\tremaining: 10.2s\n",
            "605:\ttest: 0.9407805\tbest: 0.9407854 (576)\ttotal: 15.6s\tremaining: 10.2s\n",
            "606:\ttest: 0.9407812\tbest: 0.9407854 (576)\ttotal: 15.7s\tremaining: 10.1s\n",
            "607:\ttest: 0.9407801\tbest: 0.9407854 (576)\ttotal: 15.7s\tremaining: 10.1s\n",
            "608:\ttest: 0.9407905\tbest: 0.9407905 (608)\ttotal: 15.7s\tremaining: 10.1s\n",
            "609:\ttest: 0.9407890\tbest: 0.9407905 (608)\ttotal: 15.7s\tremaining: 10.1s\n",
            "610:\ttest: 0.9407883\tbest: 0.9407905 (608)\ttotal: 15.8s\tremaining: 10s\n",
            "611:\ttest: 0.9407941\tbest: 0.9407941 (611)\ttotal: 15.8s\tremaining: 10s\n",
            "612:\ttest: 0.9407856\tbest: 0.9407941 (611)\ttotal: 15.8s\tremaining: 9.99s\n",
            "613:\ttest: 0.9407857\tbest: 0.9407941 (611)\ttotal: 15.9s\tremaining: 9.97s\n",
            "614:\ttest: 0.9407850\tbest: 0.9407941 (611)\ttotal: 15.9s\tremaining: 9.95s\n",
            "615:\ttest: 0.9407931\tbest: 0.9407941 (611)\ttotal: 15.9s\tremaining: 9.92s\n",
            "616:\ttest: 0.9407915\tbest: 0.9407941 (611)\ttotal: 15.9s\tremaining: 9.89s\n",
            "617:\ttest: 0.9407976\tbest: 0.9407976 (617)\ttotal: 16s\tremaining: 9.87s\n",
            "618:\ttest: 0.9407965\tbest: 0.9407976 (617)\ttotal: 16s\tremaining: 9.84s\n",
            "619:\ttest: 0.9407962\tbest: 0.9407976 (617)\ttotal: 16s\tremaining: 9.81s\n",
            "620:\ttest: 0.9407962\tbest: 0.9407976 (617)\ttotal: 16s\tremaining: 9.79s\n",
            "621:\ttest: 0.9408162\tbest: 0.9408162 (621)\ttotal: 16.1s\tremaining: 9.76s\n",
            "622:\ttest: 0.9408049\tbest: 0.9408162 (621)\ttotal: 16.1s\tremaining: 9.74s\n",
            "623:\ttest: 0.9408033\tbest: 0.9408162 (621)\ttotal: 16.1s\tremaining: 9.71s\n",
            "624:\ttest: 0.9408030\tbest: 0.9408162 (621)\ttotal: 16.1s\tremaining: 9.69s\n",
            "625:\ttest: 0.9408093\tbest: 0.9408162 (621)\ttotal: 16.2s\tremaining: 9.66s\n",
            "626:\ttest: 0.9408114\tbest: 0.9408162 (621)\ttotal: 16.2s\tremaining: 9.63s\n",
            "627:\ttest: 0.9408100\tbest: 0.9408162 (621)\ttotal: 16.2s\tremaining: 9.61s\n",
            "628:\ttest: 0.9408064\tbest: 0.9408162 (621)\ttotal: 16.2s\tremaining: 9.58s\n",
            "629:\ttest: 0.9408057\tbest: 0.9408162 (621)\ttotal: 16.3s\tremaining: 9.55s\n",
            "630:\ttest: 0.9408083\tbest: 0.9408162 (621)\ttotal: 16.3s\tremaining: 9.53s\n",
            "631:\ttest: 0.9408085\tbest: 0.9408162 (621)\ttotal: 16.3s\tremaining: 9.5s\n",
            "632:\ttest: 0.9408041\tbest: 0.9408162 (621)\ttotal: 16.3s\tremaining: 9.47s\n",
            "633:\ttest: 0.9408079\tbest: 0.9408162 (621)\ttotal: 16.4s\tremaining: 9.45s\n",
            "634:\ttest: 0.9408105\tbest: 0.9408162 (621)\ttotal: 16.4s\tremaining: 9.43s\n",
            "635:\ttest: 0.9408193\tbest: 0.9408193 (635)\ttotal: 16.4s\tremaining: 9.4s\n",
            "636:\ttest: 0.9408204\tbest: 0.9408204 (636)\ttotal: 16.4s\tremaining: 9.37s\n",
            "637:\ttest: 0.9408226\tbest: 0.9408226 (637)\ttotal: 16.5s\tremaining: 9.35s\n",
            "638:\ttest: 0.9408279\tbest: 0.9408279 (638)\ttotal: 16.5s\tremaining: 9.32s\n",
            "639:\ttest: 0.9408258\tbest: 0.9408279 (638)\ttotal: 16.5s\tremaining: 9.29s\n",
            "640:\ttest: 0.9408275\tbest: 0.9408279 (638)\ttotal: 16.5s\tremaining: 9.27s\n",
            "641:\ttest: 0.9408333\tbest: 0.9408333 (641)\ttotal: 16.6s\tremaining: 9.24s\n",
            "642:\ttest: 0.9408308\tbest: 0.9408333 (641)\ttotal: 16.6s\tremaining: 9.21s\n",
            "643:\ttest: 0.9408310\tbest: 0.9408333 (641)\ttotal: 16.6s\tremaining: 9.19s\n",
            "644:\ttest: 0.9408350\tbest: 0.9408350 (644)\ttotal: 16.6s\tremaining: 9.16s\n",
            "645:\ttest: 0.9408406\tbest: 0.9408406 (645)\ttotal: 16.7s\tremaining: 9.14s\n",
            "646:\ttest: 0.9408458\tbest: 0.9408458 (646)\ttotal: 16.7s\tremaining: 9.11s\n",
            "647:\ttest: 0.9408542\tbest: 0.9408542 (647)\ttotal: 16.7s\tremaining: 9.09s\n",
            "648:\ttest: 0.9408414\tbest: 0.9408542 (647)\ttotal: 16.8s\tremaining: 9.07s\n",
            "649:\ttest: 0.9408389\tbest: 0.9408542 (647)\ttotal: 16.8s\tremaining: 9.04s\n",
            "650:\ttest: 0.9408369\tbest: 0.9408542 (647)\ttotal: 16.8s\tremaining: 9.02s\n",
            "651:\ttest: 0.9408234\tbest: 0.9408542 (647)\ttotal: 16.8s\tremaining: 8.99s\n",
            "652:\ttest: 0.9408352\tbest: 0.9408542 (647)\ttotal: 16.9s\tremaining: 8.96s\n",
            "653:\ttest: 0.9408450\tbest: 0.9408542 (647)\ttotal: 16.9s\tremaining: 8.94s\n",
            "654:\ttest: 0.9408391\tbest: 0.9408542 (647)\ttotal: 16.9s\tremaining: 8.91s\n",
            "655:\ttest: 0.9408465\tbest: 0.9408542 (647)\ttotal: 16.9s\tremaining: 8.89s\n",
            "656:\ttest: 0.9408437\tbest: 0.9408542 (647)\ttotal: 17s\tremaining: 8.87s\n",
            "657:\ttest: 0.9408394\tbest: 0.9408542 (647)\ttotal: 17s\tremaining: 8.84s\n",
            "658:\ttest: 0.9408421\tbest: 0.9408542 (647)\ttotal: 17s\tremaining: 8.81s\n",
            "659:\ttest: 0.9408484\tbest: 0.9408542 (647)\ttotal: 17.1s\tremaining: 8.79s\n",
            "660:\ttest: 0.9408533\tbest: 0.9408542 (647)\ttotal: 17.1s\tremaining: 8.76s\n",
            "661:\ttest: 0.9408602\tbest: 0.9408602 (661)\ttotal: 17.1s\tremaining: 8.74s\n",
            "662:\ttest: 0.9408602\tbest: 0.9408602 (661)\ttotal: 17.1s\tremaining: 8.71s\n",
            "663:\ttest: 0.9408729\tbest: 0.9408729 (663)\ttotal: 17.2s\tremaining: 8.69s\n",
            "664:\ttest: 0.9408709\tbest: 0.9408729 (663)\ttotal: 17.2s\tremaining: 8.66s\n",
            "665:\ttest: 0.9408633\tbest: 0.9408729 (663)\ttotal: 17.2s\tremaining: 8.63s\n",
            "666:\ttest: 0.9408583\tbest: 0.9408729 (663)\ttotal: 17.2s\tremaining: 8.61s\n",
            "667:\ttest: 0.9408597\tbest: 0.9408729 (663)\ttotal: 17.3s\tremaining: 8.58s\n",
            "668:\ttest: 0.9408688\tbest: 0.9408729 (663)\ttotal: 17.3s\tremaining: 8.55s\n",
            "669:\ttest: 0.9408635\tbest: 0.9408729 (663)\ttotal: 17.3s\tremaining: 8.53s\n",
            "670:\ttest: 0.9408646\tbest: 0.9408729 (663)\ttotal: 17.3s\tremaining: 8.5s\n",
            "671:\ttest: 0.9408771\tbest: 0.9408771 (671)\ttotal: 17.4s\tremaining: 8.48s\n",
            "672:\ttest: 0.9408852\tbest: 0.9408852 (672)\ttotal: 17.4s\tremaining: 8.46s\n",
            "673:\ttest: 0.9408924\tbest: 0.9408924 (673)\ttotal: 17.4s\tremaining: 8.43s\n",
            "674:\ttest: 0.9408970\tbest: 0.9408970 (674)\ttotal: 17.5s\tremaining: 8.4s\n",
            "675:\ttest: 0.9409014\tbest: 0.9409014 (675)\ttotal: 17.5s\tremaining: 8.38s\n",
            "676:\ttest: 0.9408919\tbest: 0.9409014 (675)\ttotal: 17.5s\tremaining: 8.35s\n",
            "677:\ttest: 0.9408966\tbest: 0.9409014 (675)\ttotal: 17.5s\tremaining: 8.33s\n",
            "678:\ttest: 0.9408838\tbest: 0.9409014 (675)\ttotal: 17.6s\tremaining: 8.3s\n",
            "679:\ttest: 0.9408878\tbest: 0.9409014 (675)\ttotal: 17.6s\tremaining: 8.28s\n",
            "680:\ttest: 0.9408839\tbest: 0.9409014 (675)\ttotal: 17.6s\tremaining: 8.25s\n",
            "681:\ttest: 0.9408762\tbest: 0.9409014 (675)\ttotal: 17.6s\tremaining: 8.23s\n",
            "682:\ttest: 0.9408729\tbest: 0.9409014 (675)\ttotal: 17.7s\tremaining: 8.2s\n",
            "683:\ttest: 0.9408792\tbest: 0.9409014 (675)\ttotal: 17.7s\tremaining: 8.17s\n",
            "684:\ttest: 0.9408779\tbest: 0.9409014 (675)\ttotal: 17.7s\tremaining: 8.14s\n",
            "685:\ttest: 0.9408825\tbest: 0.9409014 (675)\ttotal: 17.7s\tremaining: 8.12s\n",
            "686:\ttest: 0.9408735\tbest: 0.9409014 (675)\ttotal: 17.8s\tremaining: 8.09s\n",
            "687:\ttest: 0.9408709\tbest: 0.9409014 (675)\ttotal: 17.8s\tremaining: 8.07s\n",
            "688:\ttest: 0.9408786\tbest: 0.9409014 (675)\ttotal: 17.8s\tremaining: 8.04s\n",
            "689:\ttest: 0.9408822\tbest: 0.9409014 (675)\ttotal: 17.8s\tremaining: 8.02s\n",
            "690:\ttest: 0.9408783\tbest: 0.9409014 (675)\ttotal: 17.9s\tremaining: 7.99s\n",
            "691:\ttest: 0.9408756\tbest: 0.9409014 (675)\ttotal: 17.9s\tremaining: 7.96s\n",
            "692:\ttest: 0.9408773\tbest: 0.9409014 (675)\ttotal: 17.9s\tremaining: 7.94s\n",
            "693:\ttest: 0.9408703\tbest: 0.9409014 (675)\ttotal: 17.9s\tremaining: 7.91s\n",
            "694:\ttest: 0.9408728\tbest: 0.9409014 (675)\ttotal: 18s\tremaining: 7.88s\n",
            "695:\ttest: 0.9408785\tbest: 0.9409014 (675)\ttotal: 18s\tremaining: 7.86s\n",
            "696:\ttest: 0.9408715\tbest: 0.9409014 (675)\ttotal: 18s\tremaining: 7.84s\n",
            "697:\ttest: 0.9408671\tbest: 0.9409014 (675)\ttotal: 18.1s\tremaining: 7.81s\n",
            "698:\ttest: 0.9408608\tbest: 0.9409014 (675)\ttotal: 18.1s\tremaining: 7.78s\n",
            "699:\ttest: 0.9408650\tbest: 0.9409014 (675)\ttotal: 18.1s\tremaining: 7.76s\n",
            "700:\ttest: 0.9408657\tbest: 0.9409014 (675)\ttotal: 18.1s\tremaining: 7.73s\n",
            "701:\ttest: 0.9408661\tbest: 0.9409014 (675)\ttotal: 18.2s\tremaining: 7.71s\n",
            "702:\ttest: 0.9408724\tbest: 0.9409014 (675)\ttotal: 18.2s\tremaining: 7.68s\n",
            "703:\ttest: 0.9408889\tbest: 0.9409014 (675)\ttotal: 18.2s\tremaining: 7.65s\n",
            "704:\ttest: 0.9408920\tbest: 0.9409014 (675)\ttotal: 18.2s\tremaining: 7.63s\n",
            "705:\ttest: 0.9408982\tbest: 0.9409014 (675)\ttotal: 18.3s\tremaining: 7.6s\n",
            "706:\ttest: 0.9408975\tbest: 0.9409014 (675)\ttotal: 18.3s\tremaining: 7.58s\n",
            "707:\ttest: 0.9408982\tbest: 0.9409014 (675)\ttotal: 18.3s\tremaining: 7.55s\n",
            "708:\ttest: 0.9408991\tbest: 0.9409014 (675)\ttotal: 18.3s\tremaining: 7.53s\n",
            "709:\ttest: 0.9409091\tbest: 0.9409091 (709)\ttotal: 18.4s\tremaining: 7.5s\n",
            "710:\ttest: 0.9409115\tbest: 0.9409115 (710)\ttotal: 18.4s\tremaining: 7.48s\n",
            "711:\ttest: 0.9409074\tbest: 0.9409115 (710)\ttotal: 18.4s\tremaining: 7.45s\n",
            "712:\ttest: 0.9409036\tbest: 0.9409115 (710)\ttotal: 18.4s\tremaining: 7.43s\n",
            "713:\ttest: 0.9409020\tbest: 0.9409115 (710)\ttotal: 18.5s\tremaining: 7.4s\n",
            "714:\ttest: 0.9409148\tbest: 0.9409148 (714)\ttotal: 18.5s\tremaining: 7.38s\n",
            "715:\ttest: 0.9409276\tbest: 0.9409276 (715)\ttotal: 18.5s\tremaining: 7.35s\n",
            "716:\ttest: 0.9409285\tbest: 0.9409285 (716)\ttotal: 18.6s\tremaining: 7.33s\n",
            "717:\ttest: 0.9409269\tbest: 0.9409285 (716)\ttotal: 18.6s\tremaining: 7.3s\n",
            "718:\ttest: 0.9409328\tbest: 0.9409328 (718)\ttotal: 18.6s\tremaining: 7.27s\n",
            "719:\ttest: 0.9409320\tbest: 0.9409328 (718)\ttotal: 18.6s\tremaining: 7.25s\n",
            "720:\ttest: 0.9409356\tbest: 0.9409356 (720)\ttotal: 18.7s\tremaining: 7.22s\n",
            "721:\ttest: 0.9409371\tbest: 0.9409371 (721)\ttotal: 18.7s\tremaining: 7.2s\n",
            "722:\ttest: 0.9409286\tbest: 0.9409371 (721)\ttotal: 18.7s\tremaining: 7.17s\n",
            "723:\ttest: 0.9409319\tbest: 0.9409371 (721)\ttotal: 18.7s\tremaining: 7.14s\n",
            "724:\ttest: 0.9409312\tbest: 0.9409371 (721)\ttotal: 18.8s\tremaining: 7.12s\n",
            "725:\ttest: 0.9409260\tbest: 0.9409371 (721)\ttotal: 18.8s\tremaining: 7.09s\n",
            "726:\ttest: 0.9409252\tbest: 0.9409371 (721)\ttotal: 18.8s\tremaining: 7.06s\n",
            "727:\ttest: 0.9409378\tbest: 0.9409378 (727)\ttotal: 18.8s\tremaining: 7.04s\n",
            "728:\ttest: 0.9409366\tbest: 0.9409378 (727)\ttotal: 18.9s\tremaining: 7.01s\n",
            "729:\ttest: 0.9409384\tbest: 0.9409384 (729)\ttotal: 18.9s\tremaining: 6.99s\n",
            "730:\ttest: 0.9409353\tbest: 0.9409384 (729)\ttotal: 18.9s\tremaining: 6.96s\n",
            "731:\ttest: 0.9409370\tbest: 0.9409384 (729)\ttotal: 18.9s\tremaining: 6.93s\n",
            "732:\ttest: 0.9409347\tbest: 0.9409384 (729)\ttotal: 19s\tremaining: 6.91s\n",
            "733:\ttest: 0.9409336\tbest: 0.9409384 (729)\ttotal: 19s\tremaining: 6.88s\n",
            "734:\ttest: 0.9409366\tbest: 0.9409384 (729)\ttotal: 19s\tremaining: 6.86s\n",
            "735:\ttest: 0.9409323\tbest: 0.9409384 (729)\ttotal: 19s\tremaining: 6.83s\n",
            "736:\ttest: 0.9409394\tbest: 0.9409394 (736)\ttotal: 19.1s\tremaining: 6.81s\n",
            "737:\ttest: 0.9409389\tbest: 0.9409394 (736)\ttotal: 19.1s\tremaining: 6.78s\n",
            "738:\ttest: 0.9409411\tbest: 0.9409411 (738)\ttotal: 19.1s\tremaining: 6.75s\n",
            "739:\ttest: 0.9409481\tbest: 0.9409481 (739)\ttotal: 19.1s\tremaining: 6.73s\n",
            "740:\ttest: 0.9409494\tbest: 0.9409494 (740)\ttotal: 19.2s\tremaining: 6.7s\n",
            "741:\ttest: 0.9409418\tbest: 0.9409494 (740)\ttotal: 19.2s\tremaining: 6.67s\n",
            "742:\ttest: 0.9409475\tbest: 0.9409494 (740)\ttotal: 19.2s\tremaining: 6.65s\n",
            "743:\ttest: 0.9409462\tbest: 0.9409494 (740)\ttotal: 19.2s\tremaining: 6.62s\n",
            "744:\ttest: 0.9409425\tbest: 0.9409494 (740)\ttotal: 19.3s\tremaining: 6.59s\n",
            "745:\ttest: 0.9409397\tbest: 0.9409494 (740)\ttotal: 19.3s\tremaining: 6.57s\n",
            "746:\ttest: 0.9409439\tbest: 0.9409494 (740)\ttotal: 19.3s\tremaining: 6.54s\n",
            "747:\ttest: 0.9409481\tbest: 0.9409494 (740)\ttotal: 19.3s\tremaining: 6.51s\n",
            "748:\ttest: 0.9409500\tbest: 0.9409500 (748)\ttotal: 19.4s\tremaining: 6.49s\n",
            "749:\ttest: 0.9409534\tbest: 0.9409534 (749)\ttotal: 19.4s\tremaining: 6.46s\n",
            "750:\ttest: 0.9409567\tbest: 0.9409567 (750)\ttotal: 19.4s\tremaining: 6.43s\n",
            "751:\ttest: 0.9409625\tbest: 0.9409625 (751)\ttotal: 19.4s\tremaining: 6.41s\n",
            "752:\ttest: 0.9409608\tbest: 0.9409625 (751)\ttotal: 19.4s\tremaining: 6.38s\n",
            "753:\ttest: 0.9409603\tbest: 0.9409625 (751)\ttotal: 19.5s\tremaining: 6.35s\n",
            "754:\ttest: 0.9409627\tbest: 0.9409627 (754)\ttotal: 19.5s\tremaining: 6.33s\n",
            "755:\ttest: 0.9409653\tbest: 0.9409653 (755)\ttotal: 19.5s\tremaining: 6.3s\n",
            "756:\ttest: 0.9409693\tbest: 0.9409693 (756)\ttotal: 19.5s\tremaining: 6.27s\n",
            "757:\ttest: 0.9409741\tbest: 0.9409741 (757)\ttotal: 19.6s\tremaining: 6.25s\n",
            "758:\ttest: 0.9409779\tbest: 0.9409779 (758)\ttotal: 19.6s\tremaining: 6.22s\n",
            "759:\ttest: 0.9409725\tbest: 0.9409779 (758)\ttotal: 19.6s\tremaining: 6.19s\n",
            "760:\ttest: 0.9409776\tbest: 0.9409779 (758)\ttotal: 19.6s\tremaining: 6.17s\n",
            "761:\ttest: 0.9409786\tbest: 0.9409786 (761)\ttotal: 19.7s\tremaining: 6.14s\n",
            "762:\ttest: 0.9409807\tbest: 0.9409807 (762)\ttotal: 19.7s\tremaining: 6.11s\n",
            "763:\ttest: 0.9409822\tbest: 0.9409822 (763)\ttotal: 19.7s\tremaining: 6.09s\n",
            "764:\ttest: 0.9409935\tbest: 0.9409935 (764)\ttotal: 19.7s\tremaining: 6.06s\n",
            "765:\ttest: 0.9409923\tbest: 0.9409935 (764)\ttotal: 19.8s\tremaining: 6.04s\n",
            "766:\ttest: 0.9409964\tbest: 0.9409964 (766)\ttotal: 19.8s\tremaining: 6.01s\n",
            "767:\ttest: 0.9409941\tbest: 0.9409964 (766)\ttotal: 19.8s\tremaining: 5.98s\n",
            "768:\ttest: 0.9409833\tbest: 0.9409964 (766)\ttotal: 19.8s\tremaining: 5.96s\n",
            "769:\ttest: 0.9409813\tbest: 0.9409964 (766)\ttotal: 19.9s\tremaining: 5.93s\n",
            "770:\ttest: 0.9409841\tbest: 0.9409964 (766)\ttotal: 19.9s\tremaining: 5.9s\n",
            "771:\ttest: 0.9409907\tbest: 0.9409964 (766)\ttotal: 19.9s\tremaining: 5.88s\n",
            "772:\ttest: 0.9409910\tbest: 0.9409964 (766)\ttotal: 19.9s\tremaining: 5.85s\n",
            "773:\ttest: 0.9409894\tbest: 0.9409964 (766)\ttotal: 20s\tremaining: 5.83s\n",
            "774:\ttest: 0.9409911\tbest: 0.9409964 (766)\ttotal: 20s\tremaining: 5.8s\n",
            "775:\ttest: 0.9409869\tbest: 0.9409964 (766)\ttotal: 20s\tremaining: 5.78s\n",
            "776:\ttest: 0.9409897\tbest: 0.9409964 (766)\ttotal: 20s\tremaining: 5.75s\n",
            "777:\ttest: 0.9409882\tbest: 0.9409964 (766)\ttotal: 20.1s\tremaining: 5.72s\n",
            "778:\ttest: 0.9409970\tbest: 0.9409970 (778)\ttotal: 20.1s\tremaining: 5.7s\n",
            "779:\ttest: 0.9409909\tbest: 0.9409970 (778)\ttotal: 20.1s\tremaining: 5.67s\n",
            "780:\ttest: 0.9409921\tbest: 0.9409970 (778)\ttotal: 20.1s\tremaining: 5.64s\n",
            "781:\ttest: 0.9409899\tbest: 0.9409970 (778)\ttotal: 20.2s\tremaining: 5.62s\n",
            "782:\ttest: 0.9409931\tbest: 0.9409970 (778)\ttotal: 20.2s\tremaining: 5.59s\n",
            "783:\ttest: 0.9409898\tbest: 0.9409970 (778)\ttotal: 20.2s\tremaining: 5.57s\n",
            "784:\ttest: 0.9409904\tbest: 0.9409970 (778)\ttotal: 20.2s\tremaining: 5.54s\n",
            "785:\ttest: 0.9409872\tbest: 0.9409970 (778)\ttotal: 20.3s\tremaining: 5.52s\n",
            "786:\ttest: 0.9409825\tbest: 0.9409970 (778)\ttotal: 20.3s\tremaining: 5.49s\n",
            "787:\ttest: 0.9409753\tbest: 0.9409970 (778)\ttotal: 20.3s\tremaining: 5.46s\n",
            "788:\ttest: 0.9409767\tbest: 0.9409970 (778)\ttotal: 20.3s\tremaining: 5.44s\n",
            "789:\ttest: 0.9409679\tbest: 0.9409970 (778)\ttotal: 20.4s\tremaining: 5.42s\n",
            "790:\ttest: 0.9409638\tbest: 0.9409970 (778)\ttotal: 20.4s\tremaining: 5.39s\n",
            "791:\ttest: 0.9409631\tbest: 0.9409970 (778)\ttotal: 20.4s\tremaining: 5.36s\n",
            "792:\ttest: 0.9409674\tbest: 0.9409970 (778)\ttotal: 20.4s\tremaining: 5.34s\n",
            "793:\ttest: 0.9409752\tbest: 0.9409970 (778)\ttotal: 20.5s\tremaining: 5.31s\n",
            "794:\ttest: 0.9409799\tbest: 0.9409970 (778)\ttotal: 20.5s\tremaining: 5.29s\n",
            "795:\ttest: 0.9409834\tbest: 0.9409970 (778)\ttotal: 20.5s\tremaining: 5.26s\n",
            "796:\ttest: 0.9409866\tbest: 0.9409970 (778)\ttotal: 20.6s\tremaining: 5.24s\n",
            "797:\ttest: 0.9409807\tbest: 0.9409970 (778)\ttotal: 20.6s\tremaining: 5.21s\n",
            "798:\ttest: 0.9409806\tbest: 0.9409970 (778)\ttotal: 20.6s\tremaining: 5.19s\n",
            "799:\ttest: 0.9409785\tbest: 0.9409970 (778)\ttotal: 20.6s\tremaining: 5.16s\n",
            "800:\ttest: 0.9409761\tbest: 0.9409970 (778)\ttotal: 20.7s\tremaining: 5.13s\n",
            "801:\ttest: 0.9409746\tbest: 0.9409970 (778)\ttotal: 20.7s\tremaining: 5.11s\n",
            "802:\ttest: 0.9409758\tbest: 0.9409970 (778)\ttotal: 20.7s\tremaining: 5.08s\n",
            "803:\ttest: 0.9409750\tbest: 0.9409970 (778)\ttotal: 20.7s\tremaining: 5.06s\n",
            "804:\ttest: 0.9409739\tbest: 0.9409970 (778)\ttotal: 20.8s\tremaining: 5.03s\n",
            "805:\ttest: 0.9409769\tbest: 0.9409970 (778)\ttotal: 20.8s\tremaining: 5.01s\n",
            "806:\ttest: 0.9409773\tbest: 0.9409970 (778)\ttotal: 20.8s\tremaining: 4.98s\n",
            "807:\ttest: 0.9409768\tbest: 0.9409970 (778)\ttotal: 20.9s\tremaining: 4.96s\n",
            "808:\ttest: 0.9409747\tbest: 0.9409970 (778)\ttotal: 20.9s\tremaining: 4.93s\n",
            "809:\ttest: 0.9409718\tbest: 0.9409970 (778)\ttotal: 20.9s\tremaining: 4.9s\n",
            "810:\ttest: 0.9409723\tbest: 0.9409970 (778)\ttotal: 20.9s\tremaining: 4.88s\n",
            "811:\ttest: 0.9409702\tbest: 0.9409970 (778)\ttotal: 21s\tremaining: 4.85s\n",
            "812:\ttest: 0.9409791\tbest: 0.9409970 (778)\ttotal: 21s\tremaining: 4.83s\n",
            "813:\ttest: 0.9409829\tbest: 0.9409970 (778)\ttotal: 21s\tremaining: 4.8s\n",
            "814:\ttest: 0.9409827\tbest: 0.9409970 (778)\ttotal: 21s\tremaining: 4.78s\n",
            "815:\ttest: 0.9409860\tbest: 0.9409970 (778)\ttotal: 21.1s\tremaining: 4.75s\n",
            "816:\ttest: 0.9409939\tbest: 0.9409970 (778)\ttotal: 21.1s\tremaining: 4.72s\n",
            "817:\ttest: 0.9409906\tbest: 0.9409970 (778)\ttotal: 21.1s\tremaining: 4.7s\n",
            "818:\ttest: 0.9409951\tbest: 0.9409970 (778)\ttotal: 21.1s\tremaining: 4.67s\n",
            "819:\ttest: 0.9410004\tbest: 0.9410004 (819)\ttotal: 21.2s\tremaining: 4.65s\n",
            "820:\ttest: 0.9409972\tbest: 0.9410004 (819)\ttotal: 21.2s\tremaining: 4.62s\n",
            "821:\ttest: 0.9409941\tbest: 0.9410004 (819)\ttotal: 21.2s\tremaining: 4.59s\n",
            "822:\ttest: 0.9409860\tbest: 0.9410004 (819)\ttotal: 21.2s\tremaining: 4.57s\n",
            "823:\ttest: 0.9409847\tbest: 0.9410004 (819)\ttotal: 21.3s\tremaining: 4.54s\n",
            "824:\ttest: 0.9409881\tbest: 0.9410004 (819)\ttotal: 21.3s\tremaining: 4.52s\n",
            "825:\ttest: 0.9409943\tbest: 0.9410004 (819)\ttotal: 21.3s\tremaining: 4.49s\n",
            "826:\ttest: 0.9410012\tbest: 0.9410012 (826)\ttotal: 21.3s\tremaining: 4.46s\n",
            "827:\ttest: 0.9410101\tbest: 0.9410101 (827)\ttotal: 21.4s\tremaining: 4.44s\n",
            "828:\ttest: 0.9410132\tbest: 0.9410132 (828)\ttotal: 21.4s\tremaining: 4.41s\n",
            "829:\ttest: 0.9410212\tbest: 0.9410212 (829)\ttotal: 21.4s\tremaining: 4.39s\n",
            "830:\ttest: 0.9410236\tbest: 0.9410236 (830)\ttotal: 21.4s\tremaining: 4.36s\n",
            "831:\ttest: 0.9410334\tbest: 0.9410334 (831)\ttotal: 21.5s\tremaining: 4.33s\n",
            "832:\ttest: 0.9410307\tbest: 0.9410334 (831)\ttotal: 21.5s\tremaining: 4.31s\n",
            "833:\ttest: 0.9410350\tbest: 0.9410350 (833)\ttotal: 21.5s\tremaining: 4.28s\n",
            "834:\ttest: 0.9410388\tbest: 0.9410388 (834)\ttotal: 21.5s\tremaining: 4.26s\n",
            "835:\ttest: 0.9410354\tbest: 0.9410388 (834)\ttotal: 21.6s\tremaining: 4.23s\n",
            "836:\ttest: 0.9410293\tbest: 0.9410388 (834)\ttotal: 21.6s\tremaining: 4.21s\n",
            "837:\ttest: 0.9410306\tbest: 0.9410388 (834)\ttotal: 21.6s\tremaining: 4.18s\n",
            "838:\ttest: 0.9410352\tbest: 0.9410388 (834)\ttotal: 21.7s\tremaining: 4.16s\n",
            "839:\ttest: 0.9410377\tbest: 0.9410388 (834)\ttotal: 21.7s\tremaining: 4.13s\n",
            "840:\ttest: 0.9410318\tbest: 0.9410388 (834)\ttotal: 21.7s\tremaining: 4.11s\n",
            "841:\ttest: 0.9410341\tbest: 0.9410388 (834)\ttotal: 21.7s\tremaining: 4.08s\n",
            "842:\ttest: 0.9410319\tbest: 0.9410388 (834)\ttotal: 21.8s\tremaining: 4.05s\n",
            "843:\ttest: 0.9410315\tbest: 0.9410388 (834)\ttotal: 21.8s\tremaining: 4.03s\n",
            "844:\ttest: 0.9410280\tbest: 0.9410388 (834)\ttotal: 21.8s\tremaining: 4s\n",
            "845:\ttest: 0.9410338\tbest: 0.9410388 (834)\ttotal: 21.8s\tremaining: 3.98s\n",
            "846:\ttest: 0.9410323\tbest: 0.9410388 (834)\ttotal: 21.9s\tremaining: 3.95s\n",
            "847:\ttest: 0.9410291\tbest: 0.9410388 (834)\ttotal: 21.9s\tremaining: 3.92s\n",
            "848:\ttest: 0.9410342\tbest: 0.9410388 (834)\ttotal: 21.9s\tremaining: 3.9s\n",
            "849:\ttest: 0.9410427\tbest: 0.9410427 (849)\ttotal: 22s\tremaining: 3.87s\n",
            "850:\ttest: 0.9410516\tbest: 0.9410516 (850)\ttotal: 22s\tremaining: 3.85s\n",
            "851:\ttest: 0.9410525\tbest: 0.9410525 (851)\ttotal: 22s\tremaining: 3.82s\n",
            "852:\ttest: 0.9410483\tbest: 0.9410525 (851)\ttotal: 22s\tremaining: 3.8s\n",
            "853:\ttest: 0.9410406\tbest: 0.9410525 (851)\ttotal: 22.1s\tremaining: 3.77s\n",
            "854:\ttest: 0.9410456\tbest: 0.9410525 (851)\ttotal: 22.1s\tremaining: 3.75s\n",
            "855:\ttest: 0.9410489\tbest: 0.9410525 (851)\ttotal: 22.1s\tremaining: 3.72s\n",
            "856:\ttest: 0.9410474\tbest: 0.9410525 (851)\ttotal: 22.1s\tremaining: 3.69s\n",
            "857:\ttest: 0.9410447\tbest: 0.9410525 (851)\ttotal: 22.2s\tremaining: 3.67s\n",
            "858:\ttest: 0.9410492\tbest: 0.9410525 (851)\ttotal: 22.2s\tremaining: 3.64s\n",
            "859:\ttest: 0.9410478\tbest: 0.9410525 (851)\ttotal: 22.2s\tremaining: 3.62s\n",
            "860:\ttest: 0.9410537\tbest: 0.9410537 (860)\ttotal: 22.2s\tremaining: 3.59s\n",
            "861:\ttest: 0.9410516\tbest: 0.9410537 (860)\ttotal: 22.3s\tremaining: 3.56s\n",
            "862:\ttest: 0.9410520\tbest: 0.9410537 (860)\ttotal: 22.3s\tremaining: 3.54s\n",
            "863:\ttest: 0.9410561\tbest: 0.9410561 (863)\ttotal: 22.3s\tremaining: 3.51s\n",
            "864:\ttest: 0.9410561\tbest: 0.9410561 (863)\ttotal: 22.3s\tremaining: 3.49s\n",
            "865:\ttest: 0.9410505\tbest: 0.9410561 (863)\ttotal: 22.4s\tremaining: 3.46s\n",
            "866:\ttest: 0.9410508\tbest: 0.9410561 (863)\ttotal: 22.4s\tremaining: 3.44s\n",
            "867:\ttest: 0.9410517\tbest: 0.9410561 (863)\ttotal: 22.4s\tremaining: 3.41s\n",
            "868:\ttest: 0.9410580\tbest: 0.9410580 (868)\ttotal: 22.4s\tremaining: 3.38s\n",
            "869:\ttest: 0.9410590\tbest: 0.9410590 (869)\ttotal: 22.5s\tremaining: 3.36s\n",
            "870:\ttest: 0.9410547\tbest: 0.9410590 (869)\ttotal: 22.5s\tremaining: 3.33s\n",
            "871:\ttest: 0.9410584\tbest: 0.9410590 (869)\ttotal: 22.5s\tremaining: 3.31s\n",
            "872:\ttest: 0.9410554\tbest: 0.9410590 (869)\ttotal: 22.6s\tremaining: 3.28s\n",
            "873:\ttest: 0.9410553\tbest: 0.9410590 (869)\ttotal: 22.6s\tremaining: 3.25s\n",
            "874:\ttest: 0.9410547\tbest: 0.9410590 (869)\ttotal: 22.6s\tremaining: 3.23s\n",
            "875:\ttest: 0.9410575\tbest: 0.9410590 (869)\ttotal: 22.6s\tremaining: 3.2s\n",
            "876:\ttest: 0.9410558\tbest: 0.9410590 (869)\ttotal: 22.6s\tremaining: 3.18s\n",
            "877:\ttest: 0.9410507\tbest: 0.9410590 (869)\ttotal: 22.7s\tremaining: 3.15s\n",
            "878:\ttest: 0.9410476\tbest: 0.9410590 (869)\ttotal: 22.7s\tremaining: 3.13s\n",
            "879:\ttest: 0.9410445\tbest: 0.9410590 (869)\ttotal: 22.7s\tremaining: 3.1s\n",
            "880:\ttest: 0.9410467\tbest: 0.9410590 (869)\ttotal: 22.8s\tremaining: 3.08s\n",
            "881:\ttest: 0.9410482\tbest: 0.9410590 (869)\ttotal: 22.8s\tremaining: 3.05s\n",
            "882:\ttest: 0.9410489\tbest: 0.9410590 (869)\ttotal: 22.8s\tremaining: 3.02s\n",
            "883:\ttest: 0.9410564\tbest: 0.9410590 (869)\ttotal: 22.9s\tremaining: 3s\n",
            "884:\ttest: 0.9410669\tbest: 0.9410669 (884)\ttotal: 22.9s\tremaining: 2.97s\n",
            "885:\ttest: 0.9410682\tbest: 0.9410682 (885)\ttotal: 22.9s\tremaining: 2.95s\n",
            "886:\ttest: 0.9410615\tbest: 0.9410682 (885)\ttotal: 22.9s\tremaining: 2.92s\n",
            "887:\ttest: 0.9410583\tbest: 0.9410682 (885)\ttotal: 23s\tremaining: 2.9s\n",
            "888:\ttest: 0.9410632\tbest: 0.9410682 (885)\ttotal: 23s\tremaining: 2.87s\n",
            "889:\ttest: 0.9410701\tbest: 0.9410701 (889)\ttotal: 23s\tremaining: 2.84s\n",
            "890:\ttest: 0.9410726\tbest: 0.9410726 (890)\ttotal: 23s\tremaining: 2.82s\n",
            "891:\ttest: 0.9410695\tbest: 0.9410726 (890)\ttotal: 23.1s\tremaining: 2.79s\n",
            "892:\ttest: 0.9410755\tbest: 0.9410755 (892)\ttotal: 23.1s\tremaining: 2.77s\n",
            "893:\ttest: 0.9410835\tbest: 0.9410835 (893)\ttotal: 23.1s\tremaining: 2.74s\n",
            "894:\ttest: 0.9410889\tbest: 0.9410889 (894)\ttotal: 23.1s\tremaining: 2.71s\n",
            "895:\ttest: 0.9410885\tbest: 0.9410889 (894)\ttotal: 23.2s\tremaining: 2.69s\n",
            "896:\ttest: 0.9410913\tbest: 0.9410913 (896)\ttotal: 23.2s\tremaining: 2.66s\n",
            "897:\ttest: 0.9410923\tbest: 0.9410923 (897)\ttotal: 23.2s\tremaining: 2.64s\n",
            "898:\ttest: 0.9410947\tbest: 0.9410947 (898)\ttotal: 23.3s\tremaining: 2.61s\n",
            "899:\ttest: 0.9410897\tbest: 0.9410947 (898)\ttotal: 23.3s\tremaining: 2.59s\n",
            "900:\ttest: 0.9410892\tbest: 0.9410947 (898)\ttotal: 23.3s\tremaining: 2.56s\n",
            "901:\ttest: 0.9410885\tbest: 0.9410947 (898)\ttotal: 23.3s\tremaining: 2.54s\n",
            "902:\ttest: 0.9410848\tbest: 0.9410947 (898)\ttotal: 23.4s\tremaining: 2.51s\n",
            "903:\ttest: 0.9410811\tbest: 0.9410947 (898)\ttotal: 23.4s\tremaining: 2.48s\n",
            "904:\ttest: 0.9410874\tbest: 0.9410947 (898)\ttotal: 23.4s\tremaining: 2.46s\n",
            "905:\ttest: 0.9410901\tbest: 0.9410947 (898)\ttotal: 23.4s\tremaining: 2.43s\n",
            "906:\ttest: 0.9410919\tbest: 0.9410947 (898)\ttotal: 23.5s\tremaining: 2.4s\n",
            "907:\ttest: 0.9410933\tbest: 0.9410947 (898)\ttotal: 23.5s\tremaining: 2.38s\n",
            "908:\ttest: 0.9410943\tbest: 0.9410947 (898)\ttotal: 23.5s\tremaining: 2.35s\n",
            "909:\ttest: 0.9410972\tbest: 0.9410972 (909)\ttotal: 23.5s\tremaining: 2.33s\n",
            "910:\ttest: 0.9410955\tbest: 0.9410972 (909)\ttotal: 23.6s\tremaining: 2.3s\n",
            "911:\ttest: 0.9410906\tbest: 0.9410972 (909)\ttotal: 23.6s\tremaining: 2.28s\n",
            "912:\ttest: 0.9410875\tbest: 0.9410972 (909)\ttotal: 23.6s\tremaining: 2.25s\n",
            "913:\ttest: 0.9410824\tbest: 0.9410972 (909)\ttotal: 23.7s\tremaining: 2.23s\n",
            "914:\ttest: 0.9410890\tbest: 0.9410972 (909)\ttotal: 23.7s\tremaining: 2.2s\n",
            "915:\ttest: 0.9410864\tbest: 0.9410972 (909)\ttotal: 23.7s\tremaining: 2.17s\n",
            "916:\ttest: 0.9410918\tbest: 0.9410972 (909)\ttotal: 23.7s\tremaining: 2.15s\n",
            "917:\ttest: 0.9410863\tbest: 0.9410972 (909)\ttotal: 23.8s\tremaining: 2.12s\n",
            "918:\ttest: 0.9410818\tbest: 0.9410972 (909)\ttotal: 23.8s\tremaining: 2.1s\n",
            "919:\ttest: 0.9410807\tbest: 0.9410972 (909)\ttotal: 23.8s\tremaining: 2.07s\n",
            "920:\ttest: 0.9410755\tbest: 0.9410972 (909)\ttotal: 23.8s\tremaining: 2.04s\n",
            "921:\ttest: 0.9410836\tbest: 0.9410972 (909)\ttotal: 23.9s\tremaining: 2.02s\n",
            "922:\ttest: 0.9410826\tbest: 0.9410972 (909)\ttotal: 23.9s\tremaining: 1.99s\n",
            "923:\ttest: 0.9410796\tbest: 0.9410972 (909)\ttotal: 23.9s\tremaining: 1.97s\n",
            "924:\ttest: 0.9410804\tbest: 0.9410972 (909)\ttotal: 23.9s\tremaining: 1.94s\n",
            "925:\ttest: 0.9410803\tbest: 0.9410972 (909)\ttotal: 24s\tremaining: 1.92s\n",
            "926:\ttest: 0.9410787\tbest: 0.9410972 (909)\ttotal: 24s\tremaining: 1.89s\n",
            "927:\ttest: 0.9410865\tbest: 0.9410972 (909)\ttotal: 24s\tremaining: 1.86s\n",
            "928:\ttest: 0.9410900\tbest: 0.9410972 (909)\ttotal: 24s\tremaining: 1.84s\n",
            "929:\ttest: 0.9410868\tbest: 0.9410972 (909)\ttotal: 24.1s\tremaining: 1.81s\n",
            "930:\ttest: 0.9410873\tbest: 0.9410972 (909)\ttotal: 24.1s\tremaining: 1.78s\n",
            "931:\ttest: 0.9410905\tbest: 0.9410972 (909)\ttotal: 24.1s\tremaining: 1.76s\n",
            "932:\ttest: 0.9410904\tbest: 0.9410972 (909)\ttotal: 24.1s\tremaining: 1.73s\n",
            "933:\ttest: 0.9410925\tbest: 0.9410972 (909)\ttotal: 24.2s\tremaining: 1.71s\n",
            "934:\ttest: 0.9410977\tbest: 0.9410977 (934)\ttotal: 24.2s\tremaining: 1.68s\n",
            "935:\ttest: 0.9410925\tbest: 0.9410977 (934)\ttotal: 24.2s\tremaining: 1.66s\n",
            "936:\ttest: 0.9411001\tbest: 0.9411001 (936)\ttotal: 24.3s\tremaining: 1.63s\n",
            "937:\ttest: 0.9411003\tbest: 0.9411003 (937)\ttotal: 24.3s\tremaining: 1.6s\n",
            "938:\ttest: 0.9410930\tbest: 0.9411003 (937)\ttotal: 24.3s\tremaining: 1.58s\n",
            "939:\ttest: 0.9410953\tbest: 0.9411003 (937)\ttotal: 24.3s\tremaining: 1.55s\n",
            "940:\ttest: 0.9410951\tbest: 0.9411003 (937)\ttotal: 24.4s\tremaining: 1.53s\n",
            "941:\ttest: 0.9410923\tbest: 0.9411003 (937)\ttotal: 24.4s\tremaining: 1.5s\n",
            "942:\ttest: 0.9410877\tbest: 0.9411003 (937)\ttotal: 24.4s\tremaining: 1.48s\n",
            "943:\ttest: 0.9410900\tbest: 0.9411003 (937)\ttotal: 24.4s\tremaining: 1.45s\n",
            "944:\ttest: 0.9410917\tbest: 0.9411003 (937)\ttotal: 24.5s\tremaining: 1.42s\n",
            "945:\ttest: 0.9410820\tbest: 0.9411003 (937)\ttotal: 24.5s\tremaining: 1.4s\n",
            "946:\ttest: 0.9410811\tbest: 0.9411003 (937)\ttotal: 24.5s\tremaining: 1.37s\n",
            "947:\ttest: 0.9410801\tbest: 0.9411003 (937)\ttotal: 24.5s\tremaining: 1.34s\n",
            "948:\ttest: 0.9410825\tbest: 0.9411003 (937)\ttotal: 24.6s\tremaining: 1.32s\n",
            "949:\ttest: 0.9410830\tbest: 0.9411003 (937)\ttotal: 24.6s\tremaining: 1.29s\n",
            "950:\ttest: 0.9410798\tbest: 0.9411003 (937)\ttotal: 24.6s\tremaining: 1.27s\n",
            "951:\ttest: 0.9410724\tbest: 0.9411003 (937)\ttotal: 24.6s\tremaining: 1.24s\n",
            "952:\ttest: 0.9410803\tbest: 0.9411003 (937)\ttotal: 24.7s\tremaining: 1.22s\n",
            "953:\ttest: 0.9410881\tbest: 0.9411003 (937)\ttotal: 24.7s\tremaining: 1.19s\n",
            "954:\ttest: 0.9410868\tbest: 0.9411003 (937)\ttotal: 24.7s\tremaining: 1.16s\n",
            "955:\ttest: 0.9410951\tbest: 0.9411003 (937)\ttotal: 24.7s\tremaining: 1.14s\n",
            "956:\ttest: 0.9410985\tbest: 0.9411003 (937)\ttotal: 24.8s\tremaining: 1.11s\n",
            "957:\ttest: 0.9410982\tbest: 0.9411003 (937)\ttotal: 24.8s\tremaining: 1.09s\n",
            "958:\ttest: 0.9410975\tbest: 0.9411003 (937)\ttotal: 24.8s\tremaining: 1.06s\n",
            "959:\ttest: 0.9411107\tbest: 0.9411107 (959)\ttotal: 24.8s\tremaining: 1.03s\n",
            "960:\ttest: 0.9411145\tbest: 0.9411145 (960)\ttotal: 24.9s\tremaining: 1.01s\n",
            "961:\ttest: 0.9411146\tbest: 0.9411146 (961)\ttotal: 24.9s\tremaining: 983ms\n",
            "962:\ttest: 0.9411099\tbest: 0.9411146 (961)\ttotal: 24.9s\tremaining: 957ms\n",
            "963:\ttest: 0.9411091\tbest: 0.9411146 (961)\ttotal: 24.9s\tremaining: 931ms\n",
            "964:\ttest: 0.9411078\tbest: 0.9411146 (961)\ttotal: 25s\tremaining: 905ms\n",
            "965:\ttest: 0.9411151\tbest: 0.9411151 (965)\ttotal: 25s\tremaining: 880ms\n",
            "966:\ttest: 0.9411167\tbest: 0.9411167 (966)\ttotal: 25s\tremaining: 854ms\n",
            "967:\ttest: 0.9411210\tbest: 0.9411210 (967)\ttotal: 25s\tremaining: 828ms\n",
            "968:\ttest: 0.9411222\tbest: 0.9411222 (968)\ttotal: 25.1s\tremaining: 802ms\n",
            "969:\ttest: 0.9411237\tbest: 0.9411237 (969)\ttotal: 25.1s\tremaining: 776ms\n",
            "970:\ttest: 0.9411208\tbest: 0.9411237 (969)\ttotal: 25.1s\tremaining: 750ms\n",
            "971:\ttest: 0.9411242\tbest: 0.9411242 (971)\ttotal: 25.1s\tremaining: 724ms\n",
            "972:\ttest: 0.9411290\tbest: 0.9411290 (972)\ttotal: 25.2s\tremaining: 698ms\n",
            "973:\ttest: 0.9411280\tbest: 0.9411290 (972)\ttotal: 25.2s\tremaining: 672ms\n",
            "974:\ttest: 0.9411267\tbest: 0.9411290 (972)\ttotal: 25.2s\tremaining: 647ms\n",
            "975:\ttest: 0.9411231\tbest: 0.9411290 (972)\ttotal: 25.2s\tremaining: 621ms\n",
            "976:\ttest: 0.9411293\tbest: 0.9411293 (976)\ttotal: 25.3s\tremaining: 595ms\n",
            "977:\ttest: 0.9411306\tbest: 0.9411306 (977)\ttotal: 25.3s\tremaining: 569ms\n",
            "978:\ttest: 0.9411250\tbest: 0.9411306 (977)\ttotal: 25.3s\tremaining: 543ms\n",
            "979:\ttest: 0.9411284\tbest: 0.9411306 (977)\ttotal: 25.3s\tremaining: 517ms\n",
            "980:\ttest: 0.9411368\tbest: 0.9411368 (980)\ttotal: 25.4s\tremaining: 491ms\n",
            "981:\ttest: 0.9411363\tbest: 0.9411368 (980)\ttotal: 25.4s\tremaining: 466ms\n",
            "982:\ttest: 0.9411394\tbest: 0.9411394 (982)\ttotal: 25.4s\tremaining: 440ms\n",
            "983:\ttest: 0.9411406\tbest: 0.9411406 (983)\ttotal: 25.5s\tremaining: 414ms\n",
            "984:\ttest: 0.9411438\tbest: 0.9411438 (984)\ttotal: 25.5s\tremaining: 388ms\n",
            "985:\ttest: 0.9411415\tbest: 0.9411438 (984)\ttotal: 25.5s\tremaining: 362ms\n",
            "986:\ttest: 0.9411452\tbest: 0.9411452 (986)\ttotal: 25.5s\tremaining: 336ms\n",
            "987:\ttest: 0.9411560\tbest: 0.9411560 (987)\ttotal: 25.6s\tremaining: 310ms\n",
            "988:\ttest: 0.9411590\tbest: 0.9411590 (988)\ttotal: 25.6s\tremaining: 285ms\n",
            "989:\ttest: 0.9411544\tbest: 0.9411590 (988)\ttotal: 25.6s\tremaining: 259ms\n",
            "990:\ttest: 0.9411588\tbest: 0.9411590 (988)\ttotal: 25.6s\tremaining: 233ms\n",
            "991:\ttest: 0.9411575\tbest: 0.9411590 (988)\ttotal: 25.7s\tremaining: 207ms\n",
            "992:\ttest: 0.9411588\tbest: 0.9411590 (988)\ttotal: 25.7s\tremaining: 181ms\n",
            "993:\ttest: 0.9411593\tbest: 0.9411593 (993)\ttotal: 25.7s\tremaining: 155ms\n",
            "994:\ttest: 0.9411581\tbest: 0.9411593 (993)\ttotal: 25.7s\tremaining: 129ms\n",
            "995:\ttest: 0.9411620\tbest: 0.9411620 (995)\ttotal: 25.8s\tremaining: 103ms\n",
            "996:\ttest: 0.9411578\tbest: 0.9411620 (995)\ttotal: 25.8s\tremaining: 77.6ms\n",
            "997:\ttest: 0.9411572\tbest: 0.9411620 (995)\ttotal: 25.8s\tremaining: 51.7ms\n",
            "998:\ttest: 0.9411629\tbest: 0.9411629 (998)\ttotal: 25.8s\tremaining: 25.9ms\n",
            "999:\ttest: 0.9411619\tbest: 0.9411629 (998)\ttotal: 25.9s\tremaining: 0us\n",
            "\n",
            "bestTest = 0.9411629468\n",
            "bestIteration = 998\n",
            "\n",
            "Shrink model to first 999 iterations.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<catboost.core.CatBoostClassifier at 0x7f0f23b9dbe0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 592
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mpvv2f_UhQeE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "test_pred = clf.predict(test_x)\n",
        "valid_pred = clf.predict(valid_x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wWKmvkLaxhW3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "test_proba = clf.predict_proba(test_x)[:, 1]\n",
        "valid_proba = clf.predict_proba(valid_x)[:, 1]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0PF8aYkExmBv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "f7145cc3-8d9a-4567-8401-960a3f617dcc"
      },
      "source": [
        "pd.Series(test_proba).hist();"
      ],
      "execution_count": 595,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAASAUlEQVR4nO3dfYxc113G8e9D0heoS5xiugqJYYPk\nIkwsSrpKg0CwViB1UqkuoooShdYuASNIEAUL1YBQqoZKRrRFqigBV7GaQukSXmslLpExXUVFBOJA\nyFspWVK3jYkSIKmLGyi4/PhjrqNhWXtnd2dnvD7fjzSaueeeO/f8vJtn7px79yZVhSSpDV8z7gFI\nkkbH0Jekhhj6ktQQQ1+SGmLoS1JDzh/3AM5kw4YNNTk5OXD/L3/5y7ziFa9YvQGdpay7HS3WDNa9\nVA8++OC/VtU3LrTurA79yclJjhw5MnD/2dlZpqenV29AZynrbkeLNYN1L1WSz51undM7ktQQQ1+S\nGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUkLP6L3JXanLPPWPZ79G9bxzLfiVpMR7p\nS1JDFg39JBuTfDLJ40keS/IzXfu7khxL8lD3uLZvm19IMpfkM0ne0Ne+rWubS7JndUqSJJ3OINM7\nJ4HdVfW3SV4JPJjkULfu16vqvf2dk2wGrge+A/gm4M+TvKZb/UHgB4GngAeSHKiqx4dRiCRpcYuG\nflU9DTzdvf73JJ8GLj7DJtuBmar6CvDZJHPAFd26uap6EiDJTNfX0JekEUlVDd45mQTuAy4Dfg7Y\nCXwJOELv28DzSX4DuL+qfrfb5g7gE91bbKuqH+va3wq8vqpumbePXcAugImJidfNzMwMPL4TJ06w\nbt26F5cfOXZ84G2HacvFF4x0f/PrbkWLdbdYM1j3Um3duvXBqppaaN3AV+8kWQf8EfCOqvpSktuB\n24Dqnt8H/OiSRzdPVe0D9gFMTU3VUu4lPf/e0zvHdfXOjdOL9hkm7zXejhZrBusepoFCP8lL6AX+\nR6vqjwGq6pm+9R8C7u4WjwEb+za/pGvjDO2SpBEY5OqdAHcAn66q9/e1X9TX7YeAR7vXB4Drk7ws\nyaXAJuBvgAeATUkuTfJSeid7DwynDEnSIAY50v8e4K3AI0ke6tp+EbghyWvpTe8cBX4CoKoeS3IX\nvRO0J4Gbq+qrAEluAe4FzgP2V9VjQ6xFkrSIQa7e+RSQBVYdPMM27wHes0D7wTNtJ0laXf5FriQ1\nxNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMM\nfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCX\npIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDVk09JNsTPLJJI8neSzJz3Ttr0pyKMkT3fOFXXuSfCDJ\nXJKHk1ze9147uv5PJNmxemVJkhYyyJH+SWB3VW0GrgRuTrIZ2AMcrqpNwOFuGeAaYFP32AXcDr0P\nCeBW4PXAFcCtpz4oJEmjsWjoV9XTVfW33et/Bz4NXAxsB+7sut0JvLl7vR34SPXcD6xPchHwBuBQ\nVT1XVc8Dh4BtQ61GknRGqarBOyeTwH3AZcDnq2p91x7g+apan+RuYG9Vfapbdxh4JzANvLyqfqVr\n/2XgP6rqvfP2sYveNwQmJiZeNzMzM/D4Tpw4wbp1615cfuTY8YG3HaYtF18w0v3Nr7sVLdbdYs1g\n3Uu1devWB6tqaqF15w/6JknWAX8EvKOqvtTL+Z6qqiSDf3qcQVXtA/YBTE1N1fT09MDbzs7O0t9/\n5557hjGkJTt64/SifYZpft2taLHuFmsG6x6mga7eSfISeoH/0ar64675mW7ahu752a79GLCxb/NL\nurbTtUuSRmSQq3cC3AF8uqre37fqAHDqCpwdwMf72t/WXcVzJXC8qp4G7gWuTnJhdwL36q5NkjQi\ng0zvfA/wVuCRJA91bb8I7AXuSnIT8Dngum7dQeBaYA54AXg7QFU9l+Q24IGu37ur6rmhVCFJGsii\nod+dkM1pVl+1QP8Cbj7Ne+0H9i9lgJKk4fEvciWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1J\naoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SG\nGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1Jasii\noZ9kf5Jnkzza1/auJMeSPNQ9ru1b9wtJ5pJ8Jskb+tq3dW1zSfYMvxRJ0mIGOdL/MLBtgfZfr6rX\ndo+DAEk2A9cD39Ft85tJzktyHvBB4BpgM3BD11eSNELnL9ahqu5LMjng+20HZqrqK8Bnk8wBV3Tr\n5qrqSYAkM13fx5c8YknSsq1kTv+WJA930z8Xdm0XA1/o6/NU13a6dknSCC16pH8atwO3AdU9vw/4\n0WEMKMkuYBfAxMQEs7OzA2974sSJ/9N/95aTwxjSki1lzMMwv+5WtFh3izWDdQ/TskK/qp459TrJ\nh4C7u8VjwMa+rpd0bZyhff577wP2AUxNTdX09PTA45qdnaW//8499wy87TAdvXF60T7DNL/uVrRY\nd4s1w3jrnhxXjux946rUvazpnSQX9S3+EHDqyp4DwPVJXpbkUmAT8DfAA8CmJJcmeSm9k70Hlj9s\nSdJyLHqkn+RjwDSwIclTwK3AdJLX0pveOQr8BEBVPZbkLnonaE8CN1fVV7v3uQW4FzgP2F9Vjw29\nGknSGQ1y9c4NCzTfcYb+7wHes0D7QeDgkkYnSRoq/yJXkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4k\nNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JD\nDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQ\nl6SGGPqS1JBFQz/J/iTPJnm0r+1VSQ4leaJ7vrBrT5IPJJlL8nCSy/u22dH1fyLJjtUpR5J0JoMc\n6X8Y2DavbQ9wuKo2AYe7ZYBrgE3dYxdwO/Q+JIBbgdcDVwC3nvqgkCSNzqKhX1X3Ac/Na94O3Nm9\nvhN4c1/7R6rnfmB9kouANwCHquq5qnoeOMT//yCRJK2yVNXinZJJ4O6quqxb/mJVre9eB3i+qtYn\nuRvYW1Wf6tYdBt4JTAMvr6pf6dp/GfiPqnrvAvvaRe9bAhMTE6+bmZkZuJgTJ06wbt26F5cfOXZ8\n4G2HacvFF4x0f/PrbkWLdbdYM4y37nHmyHLr3rp164NVNbXQuvNXOrCqqiSLf3IM/n77gH0AU1NT\nNT09PfC2s7Oz9PffueeeYQ1rSY7eOL1on2GaX3crWqy7xZphvHWPM0dWo+7lXr3zTDdtQ/f8bNd+\nDNjY1++Sru107ZKkEVpu6B8ATl2BswP4eF/727qreK4EjlfV08C9wNVJLuxO4F7dtUmSRmjR6Z0k\nH6M3J78hyVP0rsLZC9yV5Cbgc8B1XfeDwLXAHPAC8HaAqnouyW3AA12/d1fV/JPDkqRVtmjoV9UN\np1l11QJ9C7j5NO+zH9i/pNFJkobKv8iVpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0JakhK773\njs4uk+O6T8jeN45lv5KWxiN9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BAv2dRQjONS0d1bTjI9\n8r1Ka5tH+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMM\nfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhqwo9JMcTfJIkoeSHOnaXpXkUJInuucLu/Yk+UCSuSQPJ7l8\nGAVIkgY3jCP9rVX12qqa6pb3AIerahNwuFsGuAbY1D12AbcPYd+SpCVYjemd7cCd3es7gTf3tX+k\neu4H1ie5aBX2L0k6jVTV8jdOPgs8DxTw21W1L8kXq2p9tz7A81W1PsndwN6q+lS37jDwzqo6Mu89\nd9H7JsDExMTrZmZmBh7PiRMnWLdu3YvLjxw7vuzaVmLLxReMdH/9dY+r5nGY+Fp49atG+289bvN/\nx1sxzrrHmSPLrXvr1q0P9s2+/B8r/R+jf29VHUvyauBQkn/oX1lVlWRJnypVtQ/YBzA1NVXT09MD\nbzs7O0t//51j+J91Axy9cXrRPsPUX/e4ah6H3VtOct0Sfj/OBfN/x1sxzrrHmSOrUfeKpneq6lj3\n/CzwJ8AVwDOnpm2652e77seAjX2bX9K1SZJGZNmhn+QVSV556jVwNfAocADY0XXbAXy8e30AeFt3\nFc+VwPGqenrZI5ckLdlKpncmgD/pTdtzPvB7VfVnSR4A7kpyE/A54Lqu/0HgWmAOeAF4+wr2LUla\nhmWHflU9CXznAu3/Bly1QHsBNy93f5KklVvpiVypWZNjOMG3e8tJpke+V51LvA2DJDXE0Jekhhj6\nktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9J\nDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqyPnjHoC0EpN77hn3EKQ1xSN9SWqIoS9J\nDXF6R1pjxjmldXTvG8e2bw2HoS/prPfIsePs9PzNUDi9I0kNMfQlqSGGviQ1xDl9SQMb10nk3VvG\nsttz0siP9JNsS/KZJHNJ9ox6/5LUspGGfpLzgA8C1wCbgRuSbB7lGCSpZaOe3rkCmKuqJwGSzADb\ngcdHPI5VNeqvwLu3nPRyNkkDSVWNbmfJW4BtVfVj3fJbgddX1S19fXYBu7rFbwM+s4RdbAD+dUjD\nXUusux0t1gzWvVTfUlXfuNCKs+5EblXtA/YtZ9skR6pqashDOutZdztarBmse5jvOeoTuceAjX3L\nl3RtkqQRGHXoPwBsSnJpkpcC1wMHRjwGSWrWSKd3qupkkluAe4HzgP1V9dgQd7GsaaFzgHW3o8Wa\nwbqHZqQnciVJ4+VtGCSpIYa+JDVkTYb+YrdySPKyJL/frf/rJJOjH+VwDVDzzyV5PMnDSQ4n+ZZx\njHPYBr1tR5IfTlJJzonL+gapO8l13c/8sSS/N+oxroYBfs+/Ocknk/xd97t+7TjGOUxJ9id5Nsmj\np1mfJB/o/k0eTnL5inZYVWvqQe8E8D8B3wq8FPh7YPO8Pj8F/Fb3+nrg98c97hHUvBX4uu71T671\nmgetu+v3SuA+4H5gatzjHtHPexPwd8CF3fKrxz3uEdW9D/jJ7vVm4Oi4xz2Eur8PuBx49DTrrwU+\nAQS4EvjrlexvLR7pv3grh6r6L+DUrRz6bQfu7F7/IXBVkoxwjMO2aM1V9cmqeqFbvJ/e30CsdYP8\nrAFuA34V+M9RDm4VDVL3jwMfrKrnAarq2RGPcTUMUncBX9+9vgD45xGOb1VU1X3Ac2fosh34SPXc\nD6xPctFy97cWQ/9i4At9y091bQv2qaqTwHHgG0YyutUxSM39bqJ3ZLDWLVp391V3Y1WdSzcfGuTn\n/RrgNUn+Msn9SbaNbHSrZ5C63wX8SJKngIPAT49maGO11P/+z+isuw2DVibJjwBTwPePeyyrLcnX\nAO8Hdo55KONwPr0pnml63+ruS7Klqr441lGtvhuAD1fV+5J8N/A7SS6rqv8Z98DWirV4pD/IrRxe\n7JPkfHpfA/9tJKNbHQPdviLJDwC/BLypqr4yorGtpsXqfiVwGTCb5Ci9+c4D58DJ3EF+3k8BB6rq\nv6vqs8A/0vsQWMsGqfsm4C6Aqvor4OX0bkp2Lhvq7WvWYugPciuHA8CO7vVbgL+o7ozIGrVozUm+\nC/hteoF/LszvwiJ1V9XxqtpQVZNVNUnvXMabqurIeIY7NIP8jv8pvaN8kmygN93z5CgHuQoGqfvz\nwFUASb6dXuj/y0hHOXoHgLd1V/FcCRyvqqeX+2ZrbnqnTnMrhyTvBo5U1QHgDnpf++bonSC5fnwj\nXrkBa/41YB3wB905689X1ZvGNughGLDuc86Add8LXJ3kceCrwM9X1Vr+Njto3buBDyX5WXondXeu\n8QM6knyM3gf4hu5cxa3ASwCq6rfonbu4FpgDXgDevqL9rfF/L0nSEqzF6R1J0jIZ+pLUEENfkhpi\n6EtSQwx9SWqIoS9JDTH0Jakh/wsKIpjE+EAAhgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L76wDwH5h3ao",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "26765d87-432f-4145-ca2b-f8e2c79c782a"
      },
      "source": [
        "\n",
        "test_pred.mean(), test_y.mean(), (test_proba>0.42).mean()\n"
      ],
      "execution_count": 596,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.3352913897047457, 0.3705364043901685, 0.3530684804452002)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 596
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MYA1IJpOh4TX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.metrics import f1_score"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e8ShkFJLiDNg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "b68211fe-7556-4e47-93a5-98a2ae92f141"
      },
      "source": [
        "print(f1_score(test_y, test_pred))"
      ],
      "execution_count": 598,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.8098992553657468\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JxgJeYnPyM49",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "6686593f-6b8e-4200-f6af-b026a8f2f0d5"
      },
      "source": [
        "print(f1_score(test_y, test_proba>0.41))"
      ],
      "execution_count": 599,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.8193438432040904\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oaNPACGw8ARC",
        "colab_type": "text"
      },
      "source": [
        "Выбираем оптимальный порог отсечения"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6PHfAbQ9xr0P",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        },
        "outputId": "8495e1c8-aa17-484d-ec96-89e06ecf7842"
      },
      "source": [
        "thresholds = np.linspace(0.2, 0.8, num=50)\n",
        "scores = np.array([f1_score(test_y, test_proba>t) for t in thresholds])\n",
        "plt.plot(thresholds, scores);\n",
        "print(thresholds[scores.argmax()])"
      ],
      "execution_count": 570,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.40816326530612246\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXhU5d3/8fc3O9kIJIFANgIEQsJO\nAAFBrSIIKopWwa0+7m2lam2f4tM+1traWvu0at0q+rMuLSBFq1gQRAFBZAsQliQsIUBIICQhECAh\n69y/PzJojJFMksmcycz3dV25nDlzTvI5TvhwuO9zzogxBqWUUp7Lx+oASimlOpYWvVJKeTgteqWU\n8nBa9Eop5eG06JVSysNp0SullIdzqOhFZKqI7BWRXBGZ28zrCSKyWkS2i8hOEZlmXz5ZRLaKyC77\nf7/n7B1QSil1YdLSefQi4gvsAyYDBcAWYLYxJrvROvOA7caYV0QkFVhmjOkjIiOA48aYoyIyGFhh\njIntqJ1RSin1bY4c0Y8Bco0xecaYGmAhMKPJOgYItz/uChwFMMZsN8YctS/PArqISGD7YyullHKU\nnwPrxAJHGj0vAMY2WecJ4BMRmQOEAFc0831uALYZY6qbviAi9wH3AYSEhIxKSUlxIJZSSqnztm7d\nWmqMiW7uNUeK3hGzgTeNMX8WkXHAOyIy2BhjAxCRNOCPwJXNbWyMmQfMA0hPTzcZGRlOiqWUUt5B\nRA5/12uODN0UAvGNnsfZlzV2N7AIwBizAQgCouw/PA74N3CHMeaA47GVUko5gyNFvwVIFpEkEQkA\nZgFLmqyTD1wOICKDaCj6EhGJAJYCc40x650XWymllKNaLHpjTB3wILACyAEWGWOyRORJEbnWvtqj\nwL0isgNYANxpGk7neRDoDzwuIpn2rx4dsidKKaWa1eLpla6mY/RKKdV6IrLVGJPe3Gt6ZaxSSnk4\nLXqllPJwWvRKKeXhtOiVS+0uLOftDYc4VFphdRSlvIazLphS6oJKzlTzfyv2smjrEc7P/6f1Dmf6\n0F5MH9KLxMgQawMq5cG06FWHqqmz8eaXB/nrZ7lU1dZz94Qkbhodz9p9Jfxn5zGeWb6XZ5bvZXBs\nOFcP7c2tYxMIC/K3OrZSHkVPr1RtZoxh4ZYj5JdV0qtrED3Dg+jVNYiY8CAiQwNZvaeYp5blcLC0\ngu+l9OCX0wfRLzr0G9+j4GQlH+8qYumuY2QeOUV0WCBzp6Zw/YhYfHzEoj1TqvO50OmVWvSqzV5Z\nc4A/Lt+Dj4Ctya+Rr49QbzP0iw7hf69O5dKBLV8nl3nkFE8sySLzyCmGx0fwm2vTGBYf0UHplfIs\nWvTK6T7YXsjD72Zy7bDePHvzcMoqaigqr6LodBVF5ec4Vl5FbLcu3JQej7+v43P+Npvh/e2FPP3x\nHkrPVnNTehw/n5JCdJje3VqpC9GiV071ZW4pP/j7ZkYlduOtu8YQ6Ofr9J9xpqqWF1fl8sb6gwT5\n+XJxchQDeoYxMCaMAT3D6BMZjF8r/gJRytNp0SunyTl2mpv+toHeEV1Y9MA4unbp2InTvJKz/PWz\n/ewoKOfQiYqvztgJ8PWhb3QIV6bFcN+kvoQG6nkFyrtp0SunOHrqHDNf/hKA9380nt4RXVz686tq\n68ktPsu+42fYe/wMuwvLWZ97gqjQAB6+YgCzRsfrUb7yWhcqej0MUg4pP1fLnX/fTEV1Hf/64TiX\nlzxAkL8vg2O7Mji261fLMo+c4vdLc/jVB7v5+/qDzL1qEFcM6oGInrGj1Hl6+KNaVFNn44F3tnKw\ntIJXbx9FSkx4yxu5yPD4CN69/yLm3T4KY+DetzOYNW8juwvLrY6mlNvQolcXZIzhsfd3sSHvBH+6\ncRjj+0dZHelbRIQr02JY8cgkfjsjjdzis8x8+Us+2N70g9CU8k5a9OqCXl5zgPe2FfDwFclcNyLW\n6jgX5O/rw+3j+vDZo5cwIiGCh9/N5LlP9+Fu81BKuZoWvfpOy3Yd408r9jJjeG8eujzZ6jgOiwgO\n4J27xzJzZCzPfbqfny7aQXVdvdWxlLKMTsaqZu04copH3s1kVGI3/njD0E43uRng58Ofvz+MpMgQ\n/rxyH4Unz/Hq7aPoFhJgdTSlXE6P6NW3FJ46xz1vZxAdFsirt48iyN/5F0S5gogw5/Jknp81nMyC\nU8x85UsO6u2RlRfSolffcLa6jrvf3EJVTT1/v3M0UaGd/9YDM4bHMv+esZSfq+WGV76k5Ey11ZGU\nciktevWVepvhJwu2s7/4LC/dOpLknmFWR3Ka9D7dWXjfRZw+V8tfVu6zOo5SLqVFr77y3tYCVu0p\n5olrUpk0INrqOE43oGcYt49L5N0t+eQcO211HKVcRotefeWfmw4zoGcot12UaHWUDvPQ5cmEBfnz\nu6XZetql8hoOFb2ITBWRvSKSKyJzm3k9QURWi8h2EdkpItPsyyPty8+KyIvODq+cZ3dhOTsKypk9\nJqHTnWHTGhHBATxyRTLrc0/wWU6x1XGUcokWi15EfIGXgKuAVGC2iKQ2We1XwCJjzAhgFvCyfXkV\n8L/Az5yWWHWI+ZvzCfTzYeaIOKujdLhbL0qkX3QIv1+WQ02dzeo4SnU4R47oxwC5xpg8Y0wNsBCY\n0WQdA5y/AUpX4CiAMabCGPMFDYWv3NTZ6jo+3F7I1UN70zXY8z+v1d/Xh19OH0ReaQXvbDxsdRyl\nOpwjRR8LHGn0vMC+rLEngNtEpABYBsxpTQgRuU9EMkQko6SkpDWbKidYknmUipp6bhmbYHUUl7ls\nYA8mJkfx/Kf7OFlRY3UcpTqUsyZjZwNvGmPigGnAOyLi8Pc2xswzxqQbY9Kjoz3vbA93t2BzPikx\nYYxM8J7PZxURfjU9lbPVdTz3qZ5uqTybI2VcCMQ3eh5nX9bY3cAiAGPMBiAIcL/bHKpv2VVQzq7C\ncm4Z69mTsM0ZGBPG7DEJ/GNTPrnFZ6yOo1SHcaTotwDJIpIkIgE0TLYuabJOPnA5gIgMoqHodQym\nE5i/+TBB/j5uf2fKjvLTyQMI9vflqaU5VkdRqsO0WPTGmDrgQWAFkEPD2TVZIvKkiFxrX+1R4F4R\n2QEsAO409pOUReQQ8BfgThEpaOaMHWWRM1W1fJh5lGuG9iY8yPMnYZsTGRrInMv7s3pvCX9YlqN3\nuVQeyaG7VxpjltEwydp42eONHmcDE75j2z7tyKc60IeZR6n0sknY5tw5Pqnh07PW5rF2fynPzxrO\nAA+6/YNSemWslzLGMH9TPoN6hTM83nsmYZsT4OfDH2YO5bU70ik+XcXVL3zBG18cxGbTK2eVZ9Ci\n91I7C8rJPnbaKydhv8vk1J4sf3gSE/tH8eR/srnjjc0UleslIKrz06L3UvM35RMc4Mt1w3tbHcWt\nRIcF8voP0vn99UPYevgkU55by4eZhXpfHNWpadF7odNVtSzZcZRrh/UmzEsnYS9ERLhlbALLHppI\n3+gQHlqYyT1vZXCs/JzV0ZRqEy16L/TB9kLO1eokbEuSokJY/MB4fjV9EOsPlHLlX9ayYHO+Ht2r\nTkeL3svU1Nl49fM8RiREMDTOuydhHeHrI9wzsS8rHp7E4NiuPPb+Lm59fRP5JyqtjqaUw7Tovcx7\n2wooPHWOhy5PtjpKp5IYGcI/7xnL768fws6CcqY8t5Z3Nh7Wo3vVKWjRe5GaOhsvrc5leHwEl3jg\nJ0h1NB+fhrH7Tx6ZxOik7vzvB7v5+eKdVNXqRVbKvWnRe5H3txVQcPIcD12RrKdUtkPviC68eedo\nfnJ5Mou3FnDTqxs4ekonapX70qL3EjV1Nl5cncuw+Agu1aP5dvPxEX46eQCv3j6KvJIKrnnhCzbm\nnbA6llLN0qL3EueP5h/Wo3mnmpIWwwc/Hk/XLv7c9vom3lx/UMftldvRovcCtfV6NN+R+vcI44MH\nJ3DJgGie+Cibxz/MsjqSUt+gRe8Fvjqav1yP5jtKeJA/r92Rzq1jE3hn42HySs5aHUmpr2jRe7ja\nehsvrMplWFxXLh2oR/MdycdHeOiKZPx8hAWb862Oo9RXtOg93Ndj8wP0aN4FeoQFMTm1J4u3Fui9\n7ZXb0KL3YOfH5ofq0bxLzR6TwMnKWpbvLrI6ilKAFr1H+/e2Qo6U6Zk2rnZx/ygSugfr8I1yG1r0\nHqqypo7nPt3H0LiuXDawh9VxvIqPjzBrTDwb88o4oJOyyg1o0Xuol1cf4Gh5FY9fnapH8xa4cVRc\nw6TsJj2qV9bTovdAh0ormLc2j5kjYknv093qOF6pR1gQV6b15L1tBXovHGU5LXoP9OR/sgnw82Hu\nVSlWR/Fq5ydlV2TppKyylha9h/ks5zir9hTz8BXJ9AgPsjqOV5vQr2FSdr4O3yiLadF7kKraen7z\nUTb9e4Tyg/F9rI7j9c5Pym46WEZusU7KKus4VPQiMlVE9opIrojMbeb1BBFZLSLbRWSniExr9Npj\n9u32isgUZ4ZX3/Ta2jzyyyr5zbVp+Pvq3+Hu4Puj4vHzERbqqZbKQi22gYj4Ai8BVwGpwGwRSW2y\n2q+ARcaYEcAs4GX7tqn252nAVOBl+/dTTlZwspKX1uQyfUgvJvSPsjqOsosOC2RKWgyLdVJWWciR\nw74xQK4xJs8YUwMsBGY0WccA4fbHXYGj9sczgIXGmGpjzEEg1/79lJM9tTQHgP+ZPsjiJKqp2WMS\nOKWTsspCjhR9LHCk0fMC+7LGngBuE5ECYBkwpxXbIiL3iUiGiGSUlJQ4GF2dt25/CR/vLuLBy/oT\nG9HF6jiqifH9IkmMDOafOimrLOKsgdzZwJvGmDhgGvCOiDj8vY0x84wx6caY9OhovSdLaxhjePKj\nbBIjg7lnYl+r46hm+PgIs8cksPlgGev264GMcj1HyrgQiG/0PM6+rLG7gUUAxpgNQBAQ5eC2qh12\nF55mf/FZfnxpf4L8dfrDXd0xLpHkHqE8vDCT46errI6jvIwjRb8FSBaRJBEJoGFydUmTdfKBywFE\nZBANRV9iX2+WiASKSBKQDGx2VngFK7KK8PURJqf2tDqKuoDgAD9evnUklTX1/GTBdurqbVZHUl6k\nxaI3xtQBDwIrgBwazq7JEpEnReRa+2qPAveKyA5gAXCnaZBFw5F+NrAc+LExRk89cKLlWUWMTepO\nt5AAq6OoFiT3DOOp6wez6WAZz3263+o4yov4ObKSMWYZDZOsjZc93uhxNjDhO7Z9CniqHRnVd8gt\nPktu8VluvyjR6ijKQTNHxrEpr4yX1uQyOqk7l+hn+CoX0KtqOrHzp+tdmabDNp3Jb2akMbBnGI+8\nm8mx8nNWx1FeQIu+E/skq4hh8RH06qqnVHYmQf6+vHjLSKpqdbxeuYYWfSd19NQ5dhSUM0WP5jul\n/j1C+cPMIWw5dJI/r9xndRzl4bToO6lP7MM2U9NiLE6i2mrG8Fhmj0nglTUH2Jh3wuo4yoNp0XdS\ny7OKSO4RSt/oUKujqHb49TWpxIQH8Rc9qlcdSIu+EyqrqGHzwTKmDtaj+c4uyN+XH17aj80Hy/So\nXnUYLfpO6NOc49gMTNFhG49w8+h4osMC+etnem696hha9J3Qit1FxEZ0Ia13eMsrK7cX5O/L/ZP6\n8uWBE2QcKrM6jvJAWvSdzNnqOtblljIlLQYRsTqOcpJbxyYSFRrAX1flWh1FeSAt+k5mzd5iaups\nOj7vYboE+HLvxL6s3VfC9vyTVsdRHkaLvpNZkXWcyJAARiV2szqKcrLbLkqkW7A/L+hRvXIyLfpO\npLquntV7irkyrSe+Pjps42lCAv24Z2JfVu0pZldBudVxlAfRou9Evsw9wdnqOq7Us2081h3jEgkP\n8uOvq/QMHOU8WvSdyPLdRYQF+jG+X6TVUVQHCQvy5+6L+7Iy+zjZR09bHUd5CC36TqLeZliZc5zL\nUnoQ6KefJOXJ7pzQh7BAP15crUf1yjm06DsBm83w6yW7KauoYfrQXlbHUR2saxd/7pzQh2W7ithb\ndMbqOMoDaNG7uXqb4bH3d/GPjfk8cEk/rtSPDPQKd01IIjTQj98tzcYYY3Uc1clp0buxunobP//X\nDt7NOMJPLk/mF1MH6kVSXqJbSAC/uCqFdftLWbjliNVxVCenRe+mauttPLJoB+9vL+TRyQP46eQB\nWvJe5tYxCYzrG8lTS3MoPKWfRKXaToveDdXU2Zgzfzsf7TjKY1elMOfyZKsjKQv4+AjP3DgUmzHM\nfW+nDuGoNtOidzPVdfX86J9bWZ5VxONXp3L/Jf2sjqQsFN89mMemDWLd/lIWbNYhHNU2WvRu5vV1\nB/k0p5jfXjeYuy5OsjqOcgO3jklgfL9InlqaTcHJSqvjqE5Ii96N1NTZeOvLQ0xMjuL2ixKtjqPc\nhI+P8McbhgIw971dOoSjWs2hoheRqSKyV0RyRWRuM68/KyKZ9q99InKq0Wt/FJHd9q+bnRne0/xn\n51GKz1Rzz8S+VkdRbub8EM4XuaXM35xvdRzVybRY9CLiC7wEXAWkArNFJLXxOsaYR4wxw40xw4EX\ngPft204HRgLDgbHAz0REPy2jGcYYXl93kOQeoUxKjrI6jnJDt45NYEL/SH6/NEeHcFSrOHJEPwbI\nNcbkGWNqgIXAjAusPxtYYH+cCqw1xtQZYyqAncDU9gT2VBvyTpB97DR3X5ykp1GqZol8PYRz4ysb\n+M1HWXyZW0ptvc3iZMrdOVL0sUDj6f4C+7JvEZFEIAlYZV+0A5gqIsEiEgVcBsQ3s919IpIhIhkl\nJSWtye8x3vjiIJEhAVw3otn/tUoBENctmNd+kE5a73Dmb8rnltc3MfK3K/nJgobTcc9U1VodUbkh\nPyd/v1nAYmNMPYAx5hMRGQ18CZQAG4D6phsZY+YB8wDS09O9bqYpr+Qsn+YU89DlyQT56w3L1IWN\n7xfF+H5RVNbUsW5/KZ9mH+ezPcUs2XGU2IgurPrZJXrjO/UNjhzRF/LNo/A4+7LmzOLrYRsAjDFP\n2cfvJwMC7GtLUE/2xvqDBPj5cJueaaNaITjAjylpMfzp+8PY8ssreHrmEApPnWN9bqnV0ZSbcaTo\ntwDJIpIkIgE0lPmSpiuJSArQjYaj9vPLfEUk0v54KDAU+MQZwT3FyYoaFm8t4PrhsUSHBVodR3VS\nvj7CzJFxhAf58Z+dx6yOo9xMi0M3xpg6EXkQWAH4Am8YY7JE5EkgwxhzvvRnAQvNN0/y9QfW2ScX\nTwO3GWPqnLoHndz8zflU1dq4e6JeHKXaJ8DPhylpMSzfXUR1Xb0O36ivODRGb4xZBixrsuzxJs+f\naGa7KhrOvFHNaHyB1ICeYVbHUR5g2tBe/GtrAev2lXKF3tJa2emVsRbSC6SUs03oF0XXLv4s3aXD\nN+prWvQW0QukVEdoGL7pyafZx6mq/dYJbspLadFb5PwFUvdM1AuklHNNH9qbM9UNp14qBVr0llm0\n5Qjdgv2ZMVwvkFLONb5fJBHB/izdedTqKMpNaNFbwGYzrN1fymUDe+gFUsrp/H19mJoWw0odvlF2\nWvQWyD52mrKKGiYO0LF51TGmDelFRU09n+/zzluKqG/SorfA2v0Nf/gm9NeiVx1jXL9IugX7s0zP\nvlFo0Vti3b5SBvUKp0dYkNVRlIfy9/Vh6uAYPftGAVr0LldZU0fG4TI9pVJ1uOlDelNRU8+avTp8\n4+206F1sY94JausNkwZEWx1FebiL+nane0iAXjyltOhdbe2+UoL8fRiV2M3qKMrD+dmHbz7L0eEb\nb6dF72Lr9pcwNilST6tULnH1kF5U1tSzZm+x1VGUhbToXajw1DkOlFQwUcfnlYuMSepOZEiA3rrY\ny2nRu9AX9tMqdXxeucrXwzfFlJ6ttjqOsogWvQut3VdKTHgQyT1CrY6ivMjNo+OptxmmPreOz3KO\nWx1HWUCL3kXqbYYvckuZmBylNzFTLjU0LoIPH5xAVGgAd7+Vwdz3dnK2Wj//x5to0bvIrsJyys/V\nMlGHbZQFBvUK58MHJ/DAJf14N+MI055fx5ZDZVbHUi6iRe8i6/aVIAIX620PlEUC/XyZe1UKi+4f\nh8Fw06sbePrjPdTU2ayOpjqYFr2LrN1fwuDeXekeEmB1FOXlRvfpzscPTeLm9Hj+9vkBnvgoy+pI\nqoNp0bvAmapatuWfYpLerVK5idBAP56+YSj3T+rL/E35LNXTLz2aFr0LbDhwgnqbYWKyjs8r9/Kz\nKQMZkRDB3Pd2kn+i0uo4qoNo0bvAuv2lBAf4MjJBb3ug3Iu/rw9/nTUCEZizYJuO13soLXoXWLe/\nhHF9Iwnw0//dyv3Edw/mmRuHsqOgnGeW77E6juoADjWPiEwVkb0ikisic5t5/VkRybR/7RORU41e\ne0ZEskQkR0T+Kl52Enn+iUoOnajU2x4otzZ1cC/uGJfI618c1IuqPFCLRS8ivsBLwFVAKjBbRFIb\nr2OMecQYM9wYMxx4AXjfvu14YAIwFBgMjAYuceoeuLm1etsD1Un8z7RBpPYK59F/7eBY+Tmr4ygn\ncuSIfgyQa4zJM8bUAAuBGRdYfzawwP7YAEFAABAI+ANedbjw+b4SYiO6kBQVYnUUpS4oyN+XF28Z\nQU2djYcWZFJXr+P1nsKRoo8FjjR6XmBf9i0ikggkAasAjDEbgNXAMfvXCmNMTjPb3SciGSKSUVLi\nOZ+Gs//4GT7LOc5Vg2P0tgeqU+gbHcpT1w9m86Eynv9sv9VxlJM4e3ZwFrDYGFMPICL9gUFAHA1/\nOXxPRCY23cgYM88Yk26MSY+O9pwhjj98vIeQQD9+dFl/q6Mo5bDrR8Tx/VFxvLg6l/W5pVbHUU7g\nSNEXAvGNnsfZlzVnFl8P2wBcD2w0xpw1xpwFPgbGtSVoZ7M+t5RVe4p58LL+ejWs6nR+MyONftGh\nPPxuJiVn9PbGnZ0jRb8FSBaRJBEJoKHMlzRdSURSgG7AhkaL84FLRMRPRPxpmIj91tCNp6m3GZ5a\nmkNsRBd+ML6P1XGUarXgAD9evGUEp8/V8tNFmdhsxupIqh1aLHpjTB3wILCChpJeZIzJEpEnReTa\nRqvOAhYaYxr/RiwGDgC7gB3ADmPMR05L76b+vb2Q7GOn+e+pA/UjA1WnlRITzq+vSWPd/lL+tvaA\n1XFUO8g3e9l66enpJiMjw+oYbXaupp7L/m8NPcMD+fePJuDjo5OwqvMyxvDggu0s313EovsvYlRi\nd6sjqe8gIluNMenNvaaXajrZ//sij6LTVfxyeqqWvOr0RIQ/zBxCbEQX5szfzqnKmmbXq7cZquvq\nXZxOOUqL3omKz1TxypoDTEnryZgkPfJRniE8yJ8XbxlBydlqfr54J8YYjDEcLK3gnQ2HuP+dDEY8\n+QkjnlzJqj1edZlMp+FndQBP8tyn+6mus/GLqSlWR1HKqYbGRTD3qkH89j/Z3PHGZvJKKig81XD1\nbGxEF64a3IusY+Xc+/ZW/nTjUGaOjLM4sWpMi95J9h8/w8LN+dwxrg99o/XDv5XnuWtCH7YeLmN9\n7gnG94vkh5f24+L+USRGBiMinK2u4763M/jpoh2UVdRwz8S+VkdWdlr0TnL+4qifXJ5sdRSlOoSI\n8PKtozDGNHuld2igH3//r9E88m4mv1uaw4mKGv57ykC9KtwNaNE7weETFazaU8zPrhygF0cpj3eh\n4g708+WF2SOJCN7NK2sOcLKiht9dNxg/X50OtJIWvROsyCoCYMbwZm8BpJRX8fURnrpuMJEhAbyw\nKpeTlTU8P2uEXlNiIf1r1gmW7y4irXc48d2DrY6ilFsQER69ciC/viaVFVnHeeAfW6mq1dMvraJF\n307Fp6vYln+KqWkxVkdRyu3814Qknp45hDV7S7jvHS17q2jRt9Mn2Q3nDU8ZrEWvVHNmjUngmRuG\nsm5/Cfe+naFlbwEt+nZakVVEUlQIyT30lEqlvstNo+P5043D+CK3lLvf2sK5Gi17V9Kib4fyylo2\nHDjBlDT9YBGlWnLjqDj+/P1hfHngBHe9uYXKmjqrI3kNLfp2+GzPcepshilpPa2OolSnMHNkHM/e\nNJxNB7XsXUmLvh1WZBUREx7EsLgIq6Mo1WlcNyKWZ28ezuaDZdz1pg7juIIWfRudq6nn830lXJnW\nU+9SqVQrzRj+ddnrmH3H06Jvo8/3lVBVa2OKnlapVJvMGB7Ln28axoa8E3o2TgfTK2Pb6JOsIiKC\n/fV2xEq1w/Uj4rDZ4GeLd3Dv2xm8dke6Q1fQnq6q5XBpJYdOVJBfVsnZ6jouHRBNep/u+Oq/sL9F\ni74NauttfJpznMmpMfjrPTyUapcbRsVRbwy/eG8nD/xjK6/ePopAv6/LvqK6jg0HTrB2fwk7C8rJ\nL6ukrOKbH4Di5yO8suYA0WGBTE2LYdqQXoxJ0tI/T4u+DTbmneB0VR1T9SIppZzipvR4jDH84r1d\n/PAf23jkigF8kVvK2n0lZBwuo7be0MXfl+HxEUwdHENi92ASI0NIjAwmwX7rkVV7ilm26xj/2nqE\ndzYeJio0kClpPRkeH0FKTDjJPUO99n47WvRtsHx3EcEBvkxMjrI6ilIe4+bRCdTb4H/+vYtVe4oB\nSIkJ464JSUwaEE16n27fONJv6pphvblmWG8qqutYvbeh9N/fVsg/N+UD4CPQJzKEgTFhDIwJo09k\nCL0jutA7Ioie4UEe/a9zLfpWstkMn2Qf55IB0V57dKBUR7llbALx3btw/HQ1E5Oj6Bke1OrvERLo\nx9VDe3P10N7U1ds4XFbJ3qIz7Ck6w96i0+QcO83yrCKM+XobH4Ge4UH0jujCsLgIbhgVS1rvrk7c\nM2tp0bfS9iMnKTlTrcM2SnWQicnRTvtefr4+9IsOpV90KNOG9Ppq+bmaegpPneNoo6/CU1UUnKzk\nHxsP88b6gwzqFc6No+KYMbw3UaGBTstkBS36VlqRdRx/X+GylB5WR1FKtVGXAF/69wilfzP3qDpV\nWcNHO46yeGsBv/1PNn9YlsNlKT24emgv0nqH0ycypNN9kIpDRS8iU4HnAV/gdWPM001efxa4zP40\nGOhhjIkQkcuAZxutmgLMMsZ80O7kFjDGsCKriPH9oggP8rc6jlKqA0QEB3D7uD7cPq4P+4+fYfG2\nAt7fVshK+51qA/x86B8dShWki5sAAAxRSURBVEpMGCm9whgYE05KTBg9wgLd9p5XYhoPVDW3gogv\nsA+YDBQAW4DZxpjs71h/DjDCGHNXk+XdgVwgzhhT+V0/Lz093WRkZLRqJ1wl88gprntpPb+/fgi3\njE2wOo5SykXq6m3sPX7mq7H+8+P9x09Xf7VOt2B/BsaEkWIv/kG9whkS29VlV86LyFZjTHpzrzly\nRD8GyDXG5Nm/2UJgBtBs0QOzgV83s/xG4OMLlbw7M8bw1NJsuocEMH1or5Y3UEp5DD9fH9J6d/3W\nBO3JipqvSn/v8TPkHDvDoowjVNpv6ZASE8ac7yVz1eAYS2+V4kjRxwJHGj0vAMY2t6KIJAJJwKpm\nXp4F/OU7trsPuA8gIcE9j5Q/2nmMLYdO8vTMIXTtosM2SinoFhLAuH6RjOsX+dUym81QcPIcGw+e\n4G+fH+DH87cxoGcoc76XzLQhvSy5iMvZMwqzgMXGmG/ctEJEegFDgBXNbWSMmWeMSTfGpEdHO2/G\n3Vkqa+r4/dIcBseG8/30eKvjKKXcmI+PkBAZzE3p8ax85BKenzUcm4E5C7Yz5bm1fJhZSL3twkPm\nTs/kwDqFQON2i7Mva84sYEEzy28C/m2MqW1dPPfwypoDFJ2u4olr0vSSaqWUw3x9hBnDY1nx8CRe\nmD0CAR5amMktr22k9Gx1i9s7iyNFvwVIFpEkEQmgocyXNF1JRFKAbsCGZr7HbJr/C8DtHSmr5NW1\necwY3pv0PnoDM6VU6/n6CNcM682KhyfxzA1DyTxyimte+IIdR0655Oe3WPTGmDrgQRqGXXKARcaY\nLBF5UkSubbTqLGChaXIaj4j0oeFfBJ87K7Qr/W5pNn4+wmNXDbI6ilKqk/PxEW4aHc97PxyPjwjf\nf3UDizKOtLxhO7V4eqWrudPpletzS7n19U38fMpAfnxZf6vjKKU8SFlFDXMWbGN97gluvyiR/706\nlQC/tk+bXuj0ys51eZcL1dbb+M1HWSR0D+bui5OsjqOU8jDdQwJ467/GcP+kvryz8TC3vLaR4jNV\nHfKztOi/wz82Hmbf8bP8avogvXmZUqpD+Pn68Ni0QbwwewRZR09z2+ubsHXAGTl6r5tmnDhbzbMr\n9zExOYrJqT2tjqOU8nDXDOtNcs9Qys7WdMiFVVr0zXhj/UHOVtfx+NWpbnvvCqWUZ0mJCe+w761D\nN03U1ttYlFHA91J6kNwzzOo4SinVblr0TazaU0zJmWpmjXbPWzEopVRradE3sXBzPj3DA7l0oPvd\nikEppdpCi76Ro6fO8fm+Em5Kj+90HyyglFLfRduskUUZRzA0fCK9Ukp5Ci16u3qbYdGWI1zcP4r4\n7sFWx1FKKafRordbu7+Eo+VVzB6jk7BKKc+iRW+3cHM+kSEBXDFIL5BSSnkWLXqg+EwVn+UUc+Oo\nuHbdVEgppdyRthqweGsBdTbDzaN1ElYp5Xm8vuhtNsO7W44wNqk7faNDrY6jlFJO5/VFvzHvBIdP\nVOokrFLKY3l90S/YcoSuXfyZOjjG6ihKKdUhvLroyypqWLG7iOtHxOo955VSHsuri/79bQXU1Nt0\n2EYp5dG8tujrbYa3NxxmVGI3Bsbo7YiVUp7La4t++e4i8ssquXeifh6sUsqzeWXRG2N4de0BkqJC\nmJyqk7BKKc/mlUW/Ma+MnQXl3DMxCd8O+HxGpZRyJw4VvYhMFZG9IpIrInObef1ZEcm0f+0TkVON\nXksQkU9EJEdEskWkj/Pit828tQeIDAnghpFxVkdRSqkO1+KHg4uIL/ASMBkoALaIyBJjTPb5dYwx\njzRafw4wotG3eBt4yhizUkRCAZuzwrfF3qIzrN5bwqOTB+gplUopr+DIEf0YINcYk2eMqQEWAjMu\nsP5sYAGAiKQCfsaYlQDGmLPGmMp2Zm6XeWvz6OLvy20XJVoZQymlXMaRoo8FjjR6XmBf9i0ikggk\nAavsiwYAp0TkfRHZLiJ/sv8Loel294lIhohklJSUtG4PWuFY+TmW7Cjk5tHxdAsJ6LCfo5RS7sTZ\nk7GzgMXGmHr7cz9gIvAzYDTQF7iz6UbGmHnGmHRjTHp0dMd9KPff1x/CZuDui/WUSqWU93Ck6AuB\nxvfvjbMva84s7MM2dgVApn3Ypw74ABjZlqDtdbqqlvmb8pk2pJd+VKBSyqs4UvRbgGQRSRKRABrK\nfEnTlUQkBegGbGiybYSInD9M/x6Q3XRbV5i/KZ+z1XXcP6mvFT9eKaUs02LR24/EHwRWADnAImNM\nlog8KSLXNlp1FrDQGGMabVtPw7DNZyKyCxDgNWfugCNq6mz8ff1BJvSPZHBsV1f/eKWUslSLp1cC\nGGOWAcuaLHu8yfMnvmPblcDQNuZzig8zCzl+uppnbhxmZQyllLKEx18ZW1VbzytrDpASE8ak5Cir\n4yillMt5fNE/u3IfeaUVzL0qBRG93YFSyvt4dNFvPVzGvHV5zB6TwKUDe1gdRymlLOGxRV9ZU8ej\ni3YQG9GFX04fZHUcpZSyjEOTsZ3RM8v3cuhEJQvuvYjQQI/dTaWUapFHHtF/mVvKm18e4s7xfRjX\nL9LqOEopZSmPK/ozVbX8fPFOkqJC+MXUFKvjKKWU5TxuTOOppTkcKz/Hvx4YT5cAvQ2xUkp51BH9\n6r3FLNxyhPsm9WNUYjer4yillFvwmKIvr6xl7ns7GdAzlEcmJ1sdRyml3IbHDN3U1NsYEhvBQ5cn\nE+inQzZKKXWexxR9dFggr/8g3eoYSinldjxm6EYppVTztOiVUsrDadErpZSH06JXSikPp0WvlFIe\nToteKaU8nBa9Ukp5OC16pZTycGKMsTrDN4hICXC4Hd8iCih1Uhwrecp+gO6Lu/KUffGU/YD27Uui\nMSa6uRfcrujbS0QyjDGd/hJZT9kP0H1xV56yL56yH9Bx+6JDN0op5eG06JVSysN5YtHPszqAk3jK\nfoDui7vylH3xlP2ADtoXjxujV0op9U2eeESvlFKqES16pZTycJ2y6EVkqojsFZFcEZnbzOs/FZFs\nEdkpIp+JSKIVOR3hwL48ICK7RCRTRL4QkVQrcjqipX1ptN4NImJExG1PiXPgfblTRErs70umiNxj\nRc6WOPKeiMhN9j8vWSIy39UZHeXAe/Jso/djn4icsiKnIxzYlwQRWS0i2+09Nq1dP9AY06m+AF/g\nANAXCAB2AKlN1rkMCLY//iHwrtW527Ev4Y0eXwsstzp3W/fFvl4YsBbYCKRbnbsd78udwItWZ3XC\nfiQD24Fu9uc9rM7dnt+vRuvPAd6wOnc73pd5wA/tj1OBQ+35mZ3xiH4MkGuMyTPG1AALgRmNVzDG\nrDbGVNqfbgTiXJzRUY7sy+lGT0MAd509b3Ff7H4L/BGocmW4VnJ0X9ydI/txL/CSMeYkgDGm2MUZ\nHdXa92Q2sMAlyVrPkX0xQLj9cVfgaHt+YGcs+ljgSKPnBfZl3+Vu4OMOTdR2Du2LiPxYRA4AzwA/\ncVG21mpxX0RkJBBvjFnqymBt4Ojv2A32f1YvFpF410RrFUf2YwAwQETWi8hGEZnqsnSt4/Cfe/tQ\nbRKwygW52sKRfXkCuE1ECoBlNPwLpc06Y9E7TERuA9KBP1mdpT2MMS8ZY/oBvwB+ZXWethARH+Av\nwKNWZ3GSj4A+xpihwErgLYvztJUfDcM3l9JwFPyaiERYmqj9ZgGLjTH1Vgdph9nAm8aYOGAa8I79\nz1CbdMaiLwQaHz3F2Zd9g4hcAfwSuNYYU+2ibK3l0L40shC4rkMTtV1L+xIGDAbWiMgh4CJgiZtO\nyLb4vhhjTjT6vXodGOWibK3hyO9XAbDEGFNrjDkI7KOh+N1Na/6szMJ9h23AsX25G1gEYIzZAATR\ncMOztrF6YqINExl+QB4N/zQ7P5GR1mSdETRMdiRbndcJ+5Lc6PE1QIbVudu6L03WX4P7TsY68r70\navT4emCj1bnbuB9Tgbfsj6NoGFKItDp7W3+/gBTgEPaLQd3xy8H35WPgTvvjQTSM0bd5nyzf6Tb+\nj5pGw5HHAeCX9mVP0nD0DvApcBzItH8tsTpzO/bleSDLvh+rL1SeVn+1tC9N1nXbonfwffmD/X3Z\nYX9fUqzO3Mb9EBqG1LKBXcAsqzO35/eLhrHtp63O6oT3JRVYb//9ygSubM/P01sgKKWUh+uMY/RK\nKaVaQYteKaU8nBa9Ukp5OC16pZTycFr0Sinl4bTolVLKw2nRK6WUh/v/X4rfqXW6oigAAAAASUVO\nRK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qTPGtwR2kzik",
        "colab_type": "text"
      },
      "source": [
        "В ходе экспериментов я менял композицию тестовой выборки и некоторые сиды, поэтому не все числа тут сопоставимы друг с другом. Но можно видеть, что при использовании F-скора как метрики очень важно подобрать оптимальный порог. И ещё заметен последний шаг - катбуст, глядя на \"тестовую\" выборку, выбирает почему-то слишком много деревьев. Видимо, оптимумы ROC-AUC и F1 существенно разные, и деревья после тысячного уже не особо улучшают скоры около моего кастомного порога отсечения, а только где-то в хвостах распределения. \n",
        "\n",
        "* Бейзлайн - 80.3\n",
        "* Добавил тригонометрию и потюнил lr - 80.579\n",
        "* Потюнил порог - 81.47\n",
        "* Добавил логрег фичу - 81.49\n",
        "* Добавил PCA-фичи - всё только упало\n",
        "* Стоппинг по AUC = 81.76 / 80.45\n",
        "* Уменьшил LR = 81.8 / 80.87\n",
        "* Уменьшил глубину = 81.8 / 81.04\n",
        "* Текущее 81.56 / 80.36\n",
        "* RSM 0.5 =  81.46 / 80.16\n",
        "* smaller lr, more trees = 81.64 / 80.40\n",
        "* less trees = 81.93, 80.70"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pz6uL7dXmM4F",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "b4fef674-10f7-4c83-ddd2-fd96942896be"
      },
      "source": [
        "print(f1_score(valid_y, valid_pred))\n",
        "print(f1_score(valid_y, valid_proba>0.41))"
      ],
      "execution_count": 600,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.8025013027618552\n",
            "0.8070528967254408\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wp0K36-891hy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pd.options.display.max_rows = 200"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vSbmBYMu4pfn",
        "colab_type": "text"
      },
      "source": [
        "Очень важно посмотреть на важность фичей. \n",
        "\n",
        "Видим, что время звонка в очереди - суперфича. И время суток тоже. \n",
        "\n",
        "Именно поэтому выше я нарисовал гистограмки и этих фич, не поверил в увиденное, и решил применить rsm, чтобы уменьшить их влияние на итоговый прогноз. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7DE1g13GiG38",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "7c9568c9-b0b0-4040-9ddf-7a67805511eb"
      },
      "source": [
        "pd.Series(clf.feature_importances_, index=train_x.columns)"
      ],
      "execution_count": 581,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "call_start_time_abs                           2.951886\n",
              "call_end_time_abs                             3.271528\n",
              "queue_time_abs                                3.554524\n",
              "oper_start_time_abs                           5.159916\n",
              "oper_end_time_abs                             4.586468\n",
              "day_sin__call_start_time                      1.662012\n",
              "day_sin__call_end_time                        1.154483\n",
              "day_sin__queue_time                           1.836756\n",
              "day_sin__oper_start_time                      5.810892\n",
              "day_sin__oper_end_time                        7.267555\n",
              "day_cos__call_start_time                      0.596379\n",
              "day_cos__call_end_time                        0.609699\n",
              "day_cos__queue_time                           0.834917\n",
              "day_cos__oper_start_time                      1.448565\n",
              "day_cos__oper_end_time                        2.604624\n",
              "hour_sin__call_start_time                     0.284485\n",
              "hour_sin__call_end_time                       0.368895\n",
              "hour_sin__queue_time                          0.271160\n",
              "hour_sin__oper_start_time                     0.569273\n",
              "hour_sin__oper_end_time                       0.335014\n",
              "hour_cos__call_start_time                     0.405433\n",
              "hour_cos__call_end_time                       0.383292\n",
              "hour_cos__queue_time                          0.343784\n",
              "hour_cos__oper_start_time                     1.925150\n",
              "hour_cos__oper_end_time                       0.640725\n",
              "minute_sin__call_start_time                   0.490368\n",
              "minute_sin__call_end_time                     0.566943\n",
              "minute_sin__queue_time                        0.360426\n",
              "minute_sin__oper_start_time                   0.501375\n",
              "minute_sin__oper_end_time                     0.490849\n",
              "minute_cos__call_start_time                   0.326599\n",
              "minute_cos__call_end_time                     0.309607\n",
              "minute_cos__queue_time                        0.373491\n",
              "minute_cos__oper_start_time                   0.917886\n",
              "minute_cos__oper_end_time                     0.937387\n",
              "call_start_time_miss                          0.000000\n",
              "call_end_time_miss                            0.000124\n",
              "queue_time_miss                               0.228117\n",
              "oper_start_time_miss                          2.679318\n",
              "oper_end_time_miss                            1.975238\n",
              "delta_call_start_time_call_end_time           0.236829\n",
              "delta_call_start_time_queue_time              0.911058\n",
              "delta_call_start_time_oper_start_time         2.035795\n",
              "delta_call_start_time_oper_end_time           0.418269\n",
              "delta_call_end_time_queue_time                0.282645\n",
              "delta_call_end_time_oper_start_time           1.148766\n",
              "delta_call_end_time_oper_end_time             0.253337\n",
              "delta_queue_time_oper_start_time             16.919543\n",
              "delta_queue_time_oper_end_time                3.540881\n",
              "delta_oper_start_time_oper_end_time           2.122882\n",
              "rel_delta_call_start_time_call_end_time       0.002803\n",
              "rel_delta_call_start_time_queue_time          0.232344\n",
              "rel_delta_call_start_time_oper_start_time     0.930222\n",
              "rel_delta_call_start_time_oper_end_time       1.926657\n",
              "rel_delta_call_end_time_queue_time            0.313170\n",
              "rel_delta_call_end_time_oper_start_time       0.592413\n",
              "rel_delta_call_end_time_oper_end_time         0.208132\n",
              "rel_delta_queue_time_oper_start_time          2.270129\n",
              "rel_delta_queue_time_oper_end_time            0.832908\n",
              "rel_delta_oper_start_time_oper_end_time       1.953479\n",
              "oper_duration                                 1.468269\n",
              "oper_duration_miss                            1.703127\n",
              "oper_time_deviation                           0.661201\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 581
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0-yIj6ytF-3v",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "12d34a77-454d-4119-819a-b31546fd5899"
      },
      "source": [
        "best_it = clf.best_iteration_\n",
        "print(best_it)"
      ],
      "execution_count": 601,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "998\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-8ldVxIb5tlr",
        "colab_type": "text"
      },
      "source": [
        "Обучим выбранную модель на всех данных. А что, жалко же 30% выборки терять!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VfnOLc0UInZH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "all_x = extract_features(all_data)\n",
        "all_y = all_data['Метка']\n",
        "assert (all_x.columns == train_x.columns).all()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hBqXOBFRH8nk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "params = clf.get_all_params()\n",
        "params['iterations'] = clf.best_iteration_\n",
        "params['use_best_model'] = False"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Pb8kojSIt2n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "del params['bayesian_matrix_reg']\n",
        "del params['permutation_count']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ohcB9Cq1IMYd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a6aaa869-55f1-4629-cfbf-03b2b19067f9"
      },
      "source": [
        "clf2 = CatBoostClassifier(**params)\n",
        "clf2.fit(all_x, all_y, use_best_model=False)"
      ],
      "execution_count": 605,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0:\ttotal: 27.7ms\tremaining: 27.6s\n",
            "1:\ttotal: 55ms\tremaining: 27.4s\n",
            "2:\ttotal: 80.2ms\tremaining: 26.6s\n",
            "3:\ttotal: 107ms\tremaining: 26.5s\n",
            "4:\ttotal: 133ms\tremaining: 26.4s\n",
            "5:\ttotal: 158ms\tremaining: 26.2s\n",
            "6:\ttotal: 185ms\tremaining: 26.2s\n",
            "7:\ttotal: 211ms\tremaining: 26.1s\n",
            "8:\ttotal: 241ms\tremaining: 26.5s\n",
            "9:\ttotal: 267ms\tremaining: 26.4s\n",
            "10:\ttotal: 293ms\tremaining: 26.3s\n",
            "11:\ttotal: 318ms\tremaining: 26.1s\n",
            "12:\ttotal: 348ms\tremaining: 26.3s\n",
            "13:\ttotal: 374ms\tremaining: 26.3s\n",
            "14:\ttotal: 401ms\tremaining: 26.3s\n",
            "15:\ttotal: 429ms\tremaining: 26.3s\n",
            "16:\ttotal: 460ms\tremaining: 26.5s\n",
            "17:\ttotal: 486ms\tremaining: 26.5s\n",
            "18:\ttotal: 513ms\tremaining: 26.4s\n",
            "19:\ttotal: 539ms\tremaining: 26.4s\n",
            "20:\ttotal: 566ms\tremaining: 26.3s\n",
            "21:\ttotal: 592ms\tremaining: 26.3s\n",
            "22:\ttotal: 618ms\tremaining: 26.2s\n",
            "23:\ttotal: 644ms\tremaining: 26.1s\n",
            "24:\ttotal: 674ms\tremaining: 26.2s\n",
            "25:\ttotal: 701ms\tremaining: 26.2s\n",
            "26:\ttotal: 731ms\tremaining: 26.3s\n",
            "27:\ttotal: 758ms\tremaining: 26.3s\n",
            "28:\ttotal: 786ms\tremaining: 26.3s\n",
            "29:\ttotal: 819ms\tremaining: 26.4s\n",
            "30:\ttotal: 856ms\tremaining: 26.7s\n",
            "31:\ttotal: 891ms\tremaining: 26.9s\n",
            "32:\ttotal: 920ms\tremaining: 26.9s\n",
            "33:\ttotal: 949ms\tremaining: 26.9s\n",
            "34:\ttotal: 978ms\tremaining: 26.9s\n",
            "35:\ttotal: 1.01s\tremaining: 26.9s\n",
            "36:\ttotal: 1.03s\tremaining: 26.9s\n",
            "37:\ttotal: 1.06s\tremaining: 26.9s\n",
            "38:\ttotal: 1.1s\tremaining: 27.1s\n",
            "39:\ttotal: 1.13s\tremaining: 27.1s\n",
            "40:\ttotal: 1.16s\tremaining: 27.1s\n",
            "41:\ttotal: 1.19s\tremaining: 27.1s\n",
            "42:\ttotal: 1.22s\tremaining: 27.1s\n",
            "43:\ttotal: 1.25s\tremaining: 27.1s\n",
            "44:\ttotal: 1.27s\tremaining: 27s\n",
            "45:\ttotal: 1.31s\tremaining: 27.1s\n",
            "46:\ttotal: 1.33s\tremaining: 27s\n",
            "47:\ttotal: 1.36s\tremaining: 27s\n",
            "48:\ttotal: 1.39s\tremaining: 27s\n",
            "49:\ttotal: 1.42s\tremaining: 27s\n",
            "50:\ttotal: 1.45s\tremaining: 27s\n",
            "51:\ttotal: 1.48s\tremaining: 27s\n",
            "52:\ttotal: 1.52s\tremaining: 27.1s\n",
            "53:\ttotal: 1.55s\tremaining: 27.1s\n",
            "54:\ttotal: 1.58s\tremaining: 27.1s\n",
            "55:\ttotal: 1.61s\tremaining: 27.1s\n",
            "56:\ttotal: 1.64s\tremaining: 27.1s\n",
            "57:\ttotal: 1.67s\tremaining: 27.1s\n",
            "58:\ttotal: 1.7s\tremaining: 27s\n",
            "59:\ttotal: 1.73s\tremaining: 27.1s\n",
            "60:\ttotal: 1.76s\tremaining: 27.1s\n",
            "61:\ttotal: 1.8s\tremaining: 27.1s\n",
            "62:\ttotal: 1.83s\tremaining: 27.2s\n",
            "63:\ttotal: 1.89s\tremaining: 27.5s\n",
            "64:\ttotal: 1.92s\tremaining: 27.5s\n",
            "65:\ttotal: 1.95s\tremaining: 27.6s\n",
            "66:\ttotal: 1.99s\tremaining: 27.6s\n",
            "67:\ttotal: 2.02s\tremaining: 27.6s\n",
            "68:\ttotal: 2.05s\tremaining: 27.6s\n",
            "69:\ttotal: 2.07s\tremaining: 27.5s\n",
            "70:\ttotal: 2.1s\tremaining: 27.5s\n",
            "71:\ttotal: 2.14s\tremaining: 27.5s\n",
            "72:\ttotal: 2.18s\tremaining: 27.6s\n",
            "73:\ttotal: 2.21s\tremaining: 27.5s\n",
            "74:\ttotal: 2.23s\tremaining: 27.5s\n",
            "75:\ttotal: 2.26s\tremaining: 27.5s\n",
            "76:\ttotal: 2.29s\tremaining: 27.4s\n",
            "77:\ttotal: 2.33s\tremaining: 27.4s\n",
            "78:\ttotal: 2.35s\tremaining: 27.4s\n",
            "79:\ttotal: 2.39s\tremaining: 27.4s\n",
            "80:\ttotal: 2.42s\tremaining: 27.4s\n",
            "81:\ttotal: 2.45s\tremaining: 27.4s\n",
            "82:\ttotal: 2.48s\tremaining: 27.3s\n",
            "83:\ttotal: 2.51s\tremaining: 27.3s\n",
            "84:\ttotal: 2.54s\tremaining: 27.2s\n",
            "85:\ttotal: 2.56s\tremaining: 27.2s\n",
            "86:\ttotal: 2.6s\tremaining: 27.2s\n",
            "87:\ttotal: 2.63s\tremaining: 27.2s\n",
            "88:\ttotal: 2.66s\tremaining: 27.2s\n",
            "89:\ttotal: 2.69s\tremaining: 27.2s\n",
            "90:\ttotal: 2.73s\tremaining: 27.2s\n",
            "91:\ttotal: 2.75s\tremaining: 27.1s\n",
            "92:\ttotal: 2.78s\tremaining: 27.1s\n",
            "93:\ttotal: 2.82s\tremaining: 27.1s\n",
            "94:\ttotal: 2.85s\tremaining: 27.1s\n",
            "95:\ttotal: 2.88s\tremaining: 27s\n",
            "96:\ttotal: 2.9s\tremaining: 27s\n",
            "97:\ttotal: 2.94s\tremaining: 27s\n",
            "98:\ttotal: 2.97s\tremaining: 26.9s\n",
            "99:\ttotal: 3s\tremaining: 27s\n",
            "100:\ttotal: 3.04s\tremaining: 27s\n",
            "101:\ttotal: 3.07s\tremaining: 26.9s\n",
            "102:\ttotal: 3.1s\tremaining: 26.9s\n",
            "103:\ttotal: 3.13s\tremaining: 26.9s\n",
            "104:\ttotal: 3.15s\tremaining: 26.8s\n",
            "105:\ttotal: 3.19s\tremaining: 26.8s\n",
            "106:\ttotal: 3.21s\tremaining: 26.8s\n",
            "107:\ttotal: 3.25s\tremaining: 26.8s\n",
            "108:\ttotal: 3.27s\tremaining: 26.7s\n",
            "109:\ttotal: 3.3s\tremaining: 26.7s\n",
            "110:\ttotal: 3.33s\tremaining: 26.6s\n",
            "111:\ttotal: 3.36s\tremaining: 26.6s\n",
            "112:\ttotal: 3.39s\tremaining: 26.5s\n",
            "113:\ttotal: 3.42s\tremaining: 26.5s\n",
            "114:\ttotal: 3.45s\tremaining: 26.5s\n",
            "115:\ttotal: 3.48s\tremaining: 26.5s\n",
            "116:\ttotal: 3.51s\tremaining: 26.4s\n",
            "117:\ttotal: 3.54s\tremaining: 26.4s\n",
            "118:\ttotal: 3.56s\tremaining: 26.3s\n",
            "119:\ttotal: 3.59s\tremaining: 26.3s\n",
            "120:\ttotal: 3.62s\tremaining: 26.3s\n",
            "121:\ttotal: 3.65s\tremaining: 26.2s\n",
            "122:\ttotal: 3.68s\tremaining: 26.2s\n",
            "123:\ttotal: 3.71s\tremaining: 26.2s\n",
            "124:\ttotal: 3.75s\tremaining: 26.2s\n",
            "125:\ttotal: 3.78s\tremaining: 26.1s\n",
            "126:\ttotal: 3.81s\tremaining: 26.1s\n",
            "127:\ttotal: 3.83s\tremaining: 26.1s\n",
            "128:\ttotal: 3.87s\tremaining: 26.1s\n",
            "129:\ttotal: 3.9s\tremaining: 26s\n",
            "130:\ttotal: 3.93s\tremaining: 26s\n",
            "131:\ttotal: 3.96s\tremaining: 26s\n",
            "132:\ttotal: 3.99s\tremaining: 26s\n",
            "133:\ttotal: 4.02s\tremaining: 25.9s\n",
            "134:\ttotal: 4.05s\tremaining: 25.9s\n",
            "135:\ttotal: 4.09s\tremaining: 25.9s\n",
            "136:\ttotal: 4.12s\tremaining: 25.9s\n",
            "137:\ttotal: 4.15s\tremaining: 25.8s\n",
            "138:\ttotal: 4.17s\tremaining: 25.8s\n",
            "139:\ttotal: 4.2s\tremaining: 25.8s\n",
            "140:\ttotal: 4.23s\tremaining: 25.7s\n",
            "141:\ttotal: 4.26s\tremaining: 25.7s\n",
            "142:\ttotal: 4.29s\tremaining: 25.7s\n",
            "143:\ttotal: 4.32s\tremaining: 25.6s\n",
            "144:\ttotal: 4.35s\tremaining: 25.6s\n",
            "145:\ttotal: 4.38s\tremaining: 25.6s\n",
            "146:\ttotal: 4.41s\tremaining: 25.5s\n",
            "147:\ttotal: 4.44s\tremaining: 25.5s\n",
            "148:\ttotal: 4.48s\tremaining: 25.5s\n",
            "149:\ttotal: 4.51s\tremaining: 25.5s\n",
            "150:\ttotal: 4.54s\tremaining: 25.5s\n",
            "151:\ttotal: 4.57s\tremaining: 25.4s\n",
            "152:\ttotal: 4.6s\tremaining: 25.4s\n",
            "153:\ttotal: 4.63s\tremaining: 25.4s\n",
            "154:\ttotal: 4.66s\tremaining: 25.3s\n",
            "155:\ttotal: 4.68s\tremaining: 25.3s\n",
            "156:\ttotal: 4.72s\tremaining: 25.3s\n",
            "157:\ttotal: 4.76s\tremaining: 25.3s\n",
            "158:\ttotal: 4.79s\tremaining: 25.3s\n",
            "159:\ttotal: 4.83s\tremaining: 25.3s\n",
            "160:\ttotal: 4.85s\tremaining: 25.2s\n",
            "161:\ttotal: 4.88s\tremaining: 25.2s\n",
            "162:\ttotal: 4.91s\tremaining: 25.2s\n",
            "163:\ttotal: 4.95s\tremaining: 25.2s\n",
            "164:\ttotal: 4.98s\tremaining: 25.1s\n",
            "165:\ttotal: 5s\tremaining: 25.1s\n",
            "166:\ttotal: 5.03s\tremaining: 25s\n",
            "167:\ttotal: 5.06s\tremaining: 25s\n",
            "168:\ttotal: 5.09s\tremaining: 25s\n",
            "169:\ttotal: 5.12s\tremaining: 25s\n",
            "170:\ttotal: 5.16s\tremaining: 25s\n",
            "171:\ttotal: 5.19s\tremaining: 24.9s\n",
            "172:\ttotal: 5.22s\tremaining: 24.9s\n",
            "173:\ttotal: 5.25s\tremaining: 24.8s\n",
            "174:\ttotal: 5.28s\tremaining: 24.8s\n",
            "175:\ttotal: 5.3s\tremaining: 24.8s\n",
            "176:\ttotal: 5.33s\tremaining: 24.7s\n",
            "177:\ttotal: 5.37s\tremaining: 24.8s\n",
            "178:\ttotal: 5.41s\tremaining: 24.7s\n",
            "179:\ttotal: 5.43s\tremaining: 24.7s\n",
            "180:\ttotal: 5.46s\tremaining: 24.7s\n",
            "181:\ttotal: 5.49s\tremaining: 24.6s\n",
            "182:\ttotal: 5.52s\tremaining: 24.6s\n",
            "183:\ttotal: 5.55s\tremaining: 24.6s\n",
            "184:\ttotal: 5.58s\tremaining: 24.5s\n",
            "185:\ttotal: 5.61s\tremaining: 24.5s\n",
            "186:\ttotal: 5.64s\tremaining: 24.5s\n",
            "187:\ttotal: 5.67s\tremaining: 24.4s\n",
            "188:\ttotal: 5.69s\tremaining: 24.4s\n",
            "189:\ttotal: 5.72s\tremaining: 24.3s\n",
            "190:\ttotal: 5.75s\tremaining: 24.3s\n",
            "191:\ttotal: 5.78s\tremaining: 24.3s\n",
            "192:\ttotal: 5.8s\tremaining: 24.2s\n",
            "193:\ttotal: 5.83s\tremaining: 24.2s\n",
            "194:\ttotal: 5.86s\tremaining: 24.1s\n",
            "195:\ttotal: 5.88s\tremaining: 24.1s\n",
            "196:\ttotal: 5.91s\tremaining: 24s\n",
            "197:\ttotal: 5.94s\tremaining: 24s\n",
            "198:\ttotal: 5.97s\tremaining: 24s\n",
            "199:\ttotal: 5.99s\tremaining: 23.9s\n",
            "200:\ttotal: 6.03s\tremaining: 23.9s\n",
            "201:\ttotal: 6.05s\tremaining: 23.9s\n",
            "202:\ttotal: 6.08s\tremaining: 23.8s\n",
            "203:\ttotal: 6.11s\tremaining: 23.8s\n",
            "204:\ttotal: 6.13s\tremaining: 23.7s\n",
            "205:\ttotal: 6.16s\tremaining: 23.7s\n",
            "206:\ttotal: 6.19s\tremaining: 23.6s\n",
            "207:\ttotal: 6.21s\tremaining: 23.6s\n",
            "208:\ttotal: 6.24s\tremaining: 23.6s\n",
            "209:\ttotal: 6.27s\tremaining: 23.5s\n",
            "210:\ttotal: 6.3s\tremaining: 23.5s\n",
            "211:\ttotal: 6.32s\tremaining: 23.4s\n",
            "212:\ttotal: 6.35s\tremaining: 23.4s\n",
            "213:\ttotal: 6.37s\tremaining: 23.4s\n",
            "214:\ttotal: 6.4s\tremaining: 23.3s\n",
            "215:\ttotal: 6.43s\tremaining: 23.3s\n",
            "216:\ttotal: 6.47s\tremaining: 23.3s\n",
            "217:\ttotal: 6.5s\tremaining: 23.2s\n",
            "218:\ttotal: 6.52s\tremaining: 23.2s\n",
            "219:\ttotal: 6.55s\tremaining: 23.2s\n",
            "220:\ttotal: 6.58s\tremaining: 23.1s\n",
            "221:\ttotal: 6.6s\tremaining: 23.1s\n",
            "222:\ttotal: 6.63s\tremaining: 23s\n",
            "223:\ttotal: 6.66s\tremaining: 23s\n",
            "224:\ttotal: 6.69s\tremaining: 23s\n",
            "225:\ttotal: 6.71s\tremaining: 22.9s\n",
            "226:\ttotal: 6.74s\tremaining: 22.9s\n",
            "227:\ttotal: 6.77s\tremaining: 22.9s\n",
            "228:\ttotal: 6.79s\tremaining: 22.8s\n",
            "229:\ttotal: 6.82s\tremaining: 22.8s\n",
            "230:\ttotal: 6.85s\tremaining: 22.7s\n",
            "231:\ttotal: 6.87s\tremaining: 22.7s\n",
            "232:\ttotal: 6.91s\tremaining: 22.7s\n",
            "233:\ttotal: 6.93s\tremaining: 22.6s\n",
            "234:\ttotal: 6.96s\tremaining: 22.6s\n",
            "235:\ttotal: 6.99s\tremaining: 22.6s\n",
            "236:\ttotal: 7.01s\tremaining: 22.5s\n",
            "237:\ttotal: 7.04s\tremaining: 22.5s\n",
            "238:\ttotal: 7.06s\tremaining: 22.4s\n",
            "239:\ttotal: 7.09s\tremaining: 22.4s\n",
            "240:\ttotal: 7.12s\tremaining: 22.4s\n",
            "241:\ttotal: 7.16s\tremaining: 22.4s\n",
            "242:\ttotal: 7.19s\tremaining: 22.3s\n",
            "243:\ttotal: 7.21s\tremaining: 22.3s\n",
            "244:\ttotal: 7.24s\tremaining: 22.3s\n",
            "245:\ttotal: 7.27s\tremaining: 22.2s\n",
            "246:\ttotal: 7.3s\tremaining: 22.2s\n",
            "247:\ttotal: 7.33s\tremaining: 22.2s\n",
            "248:\ttotal: 7.35s\tremaining: 22.1s\n",
            "249:\ttotal: 7.38s\tremaining: 22.1s\n",
            "250:\ttotal: 7.41s\tremaining: 22s\n",
            "251:\ttotal: 7.44s\tremaining: 22s\n",
            "252:\ttotal: 7.46s\tremaining: 22s\n",
            "253:\ttotal: 7.49s\tremaining: 21.9s\n",
            "254:\ttotal: 7.52s\tremaining: 21.9s\n",
            "255:\ttotal: 7.55s\tremaining: 21.9s\n",
            "256:\ttotal: 7.58s\tremaining: 21.8s\n",
            "257:\ttotal: 7.6s\tremaining: 21.8s\n",
            "258:\ttotal: 7.63s\tremaining: 21.8s\n",
            "259:\ttotal: 7.65s\tremaining: 21.7s\n",
            "260:\ttotal: 7.68s\tremaining: 21.7s\n",
            "261:\ttotal: 7.71s\tremaining: 21.6s\n",
            "262:\ttotal: 7.73s\tremaining: 21.6s\n",
            "263:\ttotal: 7.77s\tremaining: 21.6s\n",
            "264:\ttotal: 7.79s\tremaining: 21.6s\n",
            "265:\ttotal: 7.82s\tremaining: 21.5s\n",
            "266:\ttotal: 7.84s\tremaining: 21.5s\n",
            "267:\ttotal: 7.87s\tremaining: 21.4s\n",
            "268:\ttotal: 7.9s\tremaining: 21.4s\n",
            "269:\ttotal: 7.92s\tremaining: 21.4s\n",
            "270:\ttotal: 7.95s\tremaining: 21.3s\n",
            "271:\ttotal: 7.98s\tremaining: 21.3s\n",
            "272:\ttotal: 8.01s\tremaining: 21.3s\n",
            "273:\ttotal: 8.04s\tremaining: 21.2s\n",
            "274:\ttotal: 8.06s\tremaining: 21.2s\n",
            "275:\ttotal: 8.09s\tremaining: 21.2s\n",
            "276:\ttotal: 8.12s\tremaining: 21.1s\n",
            "277:\ttotal: 8.14s\tremaining: 21.1s\n",
            "278:\ttotal: 8.17s\tremaining: 21.1s\n",
            "279:\ttotal: 8.21s\tremaining: 21.1s\n",
            "280:\ttotal: 8.24s\tremaining: 21s\n",
            "281:\ttotal: 8.27s\tremaining: 21s\n",
            "282:\ttotal: 8.29s\tremaining: 20.9s\n",
            "283:\ttotal: 8.32s\tremaining: 20.9s\n",
            "284:\ttotal: 8.34s\tremaining: 20.9s\n",
            "285:\ttotal: 8.37s\tremaining: 20.8s\n",
            "286:\ttotal: 8.4s\tremaining: 20.8s\n",
            "287:\ttotal: 8.43s\tremaining: 20.8s\n",
            "288:\ttotal: 8.46s\tremaining: 20.8s\n",
            "289:\ttotal: 8.49s\tremaining: 20.7s\n",
            "290:\ttotal: 8.51s\tremaining: 20.7s\n",
            "291:\ttotal: 8.54s\tremaining: 20.6s\n",
            "292:\ttotal: 8.57s\tremaining: 20.6s\n",
            "293:\ttotal: 8.59s\tremaining: 20.6s\n",
            "294:\ttotal: 8.62s\tremaining: 20.5s\n",
            "295:\ttotal: 8.65s\tremaining: 20.5s\n",
            "296:\ttotal: 8.68s\tremaining: 20.5s\n",
            "297:\ttotal: 8.7s\tremaining: 20.4s\n",
            "298:\ttotal: 8.73s\tremaining: 20.4s\n",
            "299:\ttotal: 8.76s\tremaining: 20.4s\n",
            "300:\ttotal: 8.79s\tremaining: 20.3s\n",
            "301:\ttotal: 8.82s\tremaining: 20.3s\n",
            "302:\ttotal: 8.84s\tremaining: 20.3s\n",
            "303:\ttotal: 8.88s\tremaining: 20.3s\n",
            "304:\ttotal: 8.9s\tremaining: 20.2s\n",
            "305:\ttotal: 8.94s\tremaining: 20.2s\n",
            "306:\ttotal: 8.97s\tremaining: 20.2s\n",
            "307:\ttotal: 8.99s\tremaining: 20.1s\n",
            "308:\ttotal: 9.02s\tremaining: 20.1s\n",
            "309:\ttotal: 9.05s\tremaining: 20.1s\n",
            "310:\ttotal: 9.08s\tremaining: 20.1s\n",
            "311:\ttotal: 9.11s\tremaining: 20s\n",
            "312:\ttotal: 9.13s\tremaining: 20s\n",
            "313:\ttotal: 9.16s\tremaining: 20s\n",
            "314:\ttotal: 9.19s\tremaining: 19.9s\n",
            "315:\ttotal: 9.21s\tremaining: 19.9s\n",
            "316:\ttotal: 9.24s\tremaining: 19.9s\n",
            "317:\ttotal: 9.27s\tremaining: 19.8s\n",
            "318:\ttotal: 9.3s\tremaining: 19.8s\n",
            "319:\ttotal: 9.33s\tremaining: 19.8s\n",
            "320:\ttotal: 9.36s\tremaining: 19.7s\n",
            "321:\ttotal: 9.38s\tremaining: 19.7s\n",
            "322:\ttotal: 9.41s\tremaining: 19.7s\n",
            "323:\ttotal: 9.43s\tremaining: 19.6s\n",
            "324:\ttotal: 9.46s\tremaining: 19.6s\n",
            "325:\ttotal: 9.49s\tremaining: 19.6s\n",
            "326:\ttotal: 9.52s\tremaining: 19.5s\n",
            "327:\ttotal: 9.55s\tremaining: 19.5s\n",
            "328:\ttotal: 9.57s\tremaining: 19.5s\n",
            "329:\ttotal: 9.6s\tremaining: 19.4s\n",
            "330:\ttotal: 9.63s\tremaining: 19.4s\n",
            "331:\ttotal: 9.65s\tremaining: 19.4s\n",
            "332:\ttotal: 9.68s\tremaining: 19.3s\n",
            "333:\ttotal: 9.71s\tremaining: 19.3s\n",
            "334:\ttotal: 9.74s\tremaining: 19.3s\n",
            "335:\ttotal: 9.77s\tremaining: 19.2s\n",
            "336:\ttotal: 9.79s\tremaining: 19.2s\n",
            "337:\ttotal: 9.82s\tremaining: 19.2s\n",
            "338:\ttotal: 9.85s\tremaining: 19.1s\n",
            "339:\ttotal: 9.87s\tremaining: 19.1s\n",
            "340:\ttotal: 9.9s\tremaining: 19.1s\n",
            "341:\ttotal: 9.93s\tremaining: 19s\n",
            "342:\ttotal: 9.95s\tremaining: 19s\n",
            "343:\ttotal: 9.98s\tremaining: 19s\n",
            "344:\ttotal: 10s\tremaining: 18.9s\n",
            "345:\ttotal: 10s\tremaining: 18.9s\n",
            "346:\ttotal: 10.1s\tremaining: 18.9s\n",
            "347:\ttotal: 10.1s\tremaining: 18.9s\n",
            "348:\ttotal: 10.1s\tremaining: 18.8s\n",
            "349:\ttotal: 10.1s\tremaining: 18.8s\n",
            "350:\ttotal: 10.2s\tremaining: 18.8s\n",
            "351:\ttotal: 10.2s\tremaining: 18.7s\n",
            "352:\ttotal: 10.2s\tremaining: 18.7s\n",
            "353:\ttotal: 10.3s\tremaining: 18.7s\n",
            "354:\ttotal: 10.3s\tremaining: 18.6s\n",
            "355:\ttotal: 10.3s\tremaining: 18.6s\n",
            "356:\ttotal: 10.3s\tremaining: 18.6s\n",
            "357:\ttotal: 10.4s\tremaining: 18.5s\n",
            "358:\ttotal: 10.4s\tremaining: 18.5s\n",
            "359:\ttotal: 10.4s\tremaining: 18.5s\n",
            "360:\ttotal: 10.5s\tremaining: 18.4s\n",
            "361:\ttotal: 10.5s\tremaining: 18.4s\n",
            "362:\ttotal: 10.5s\tremaining: 18.4s\n",
            "363:\ttotal: 10.5s\tremaining: 18.3s\n",
            "364:\ttotal: 10.6s\tremaining: 18.3s\n",
            "365:\ttotal: 10.6s\tremaining: 18.3s\n",
            "366:\ttotal: 10.6s\tremaining: 18.3s\n",
            "367:\ttotal: 10.6s\tremaining: 18.2s\n",
            "368:\ttotal: 10.7s\tremaining: 18.2s\n",
            "369:\ttotal: 10.7s\tremaining: 18.2s\n",
            "370:\ttotal: 10.7s\tremaining: 18.1s\n",
            "371:\ttotal: 10.7s\tremaining: 18.1s\n",
            "372:\ttotal: 10.8s\tremaining: 18.1s\n",
            "373:\ttotal: 10.8s\tremaining: 18s\n",
            "374:\ttotal: 10.8s\tremaining: 18s\n",
            "375:\ttotal: 10.9s\tremaining: 18s\n",
            "376:\ttotal: 10.9s\tremaining: 17.9s\n",
            "377:\ttotal: 10.9s\tremaining: 17.9s\n",
            "378:\ttotal: 10.9s\tremaining: 17.9s\n",
            "379:\ttotal: 11s\tremaining: 17.8s\n",
            "380:\ttotal: 11s\tremaining: 17.8s\n",
            "381:\ttotal: 11s\tremaining: 17.8s\n",
            "382:\ttotal: 11.1s\tremaining: 17.7s\n",
            "383:\ttotal: 11.1s\tremaining: 17.7s\n",
            "384:\ttotal: 11.1s\tremaining: 17.7s\n",
            "385:\ttotal: 11.1s\tremaining: 17.7s\n",
            "386:\ttotal: 11.2s\tremaining: 17.6s\n",
            "387:\ttotal: 11.2s\tremaining: 17.6s\n",
            "388:\ttotal: 11.2s\tremaining: 17.6s\n",
            "389:\ttotal: 11.2s\tremaining: 17.5s\n",
            "390:\ttotal: 11.3s\tremaining: 17.5s\n",
            "391:\ttotal: 11.3s\tremaining: 17.5s\n",
            "392:\ttotal: 11.3s\tremaining: 17.4s\n",
            "393:\ttotal: 11.4s\tremaining: 17.4s\n",
            "394:\ttotal: 11.4s\tremaining: 17.4s\n",
            "395:\ttotal: 11.4s\tremaining: 17.4s\n",
            "396:\ttotal: 11.4s\tremaining: 17.3s\n",
            "397:\ttotal: 11.5s\tremaining: 17.3s\n",
            "398:\ttotal: 11.5s\tremaining: 17.3s\n",
            "399:\ttotal: 11.5s\tremaining: 17.2s\n",
            "400:\ttotal: 11.6s\tremaining: 17.2s\n",
            "401:\ttotal: 11.6s\tremaining: 17.2s\n",
            "402:\ttotal: 11.6s\tremaining: 17.1s\n",
            "403:\ttotal: 11.6s\tremaining: 17.1s\n",
            "404:\ttotal: 11.7s\tremaining: 17.1s\n",
            "405:\ttotal: 11.7s\tremaining: 17.1s\n",
            "406:\ttotal: 11.7s\tremaining: 17s\n",
            "407:\ttotal: 11.8s\tremaining: 17s\n",
            "408:\ttotal: 11.8s\tremaining: 17s\n",
            "409:\ttotal: 11.8s\tremaining: 17s\n",
            "410:\ttotal: 11.9s\tremaining: 16.9s\n",
            "411:\ttotal: 11.9s\tremaining: 16.9s\n",
            "412:\ttotal: 11.9s\tremaining: 16.9s\n",
            "413:\ttotal: 11.9s\tremaining: 16.9s\n",
            "414:\ttotal: 12s\tremaining: 16.8s\n",
            "415:\ttotal: 12s\tremaining: 16.8s\n",
            "416:\ttotal: 12s\tremaining: 16.8s\n",
            "417:\ttotal: 12.1s\tremaining: 16.7s\n",
            "418:\ttotal: 12.1s\tremaining: 16.7s\n",
            "419:\ttotal: 12.1s\tremaining: 16.7s\n",
            "420:\ttotal: 12.2s\tremaining: 16.7s\n",
            "421:\ttotal: 12.2s\tremaining: 16.6s\n",
            "422:\ttotal: 12.2s\tremaining: 16.6s\n",
            "423:\ttotal: 12.2s\tremaining: 16.6s\n",
            "424:\ttotal: 12.3s\tremaining: 16.6s\n",
            "425:\ttotal: 12.3s\tremaining: 16.5s\n",
            "426:\ttotal: 12.3s\tremaining: 16.5s\n",
            "427:\ttotal: 12.4s\tremaining: 16.5s\n",
            "428:\ttotal: 12.4s\tremaining: 16.4s\n",
            "429:\ttotal: 12.4s\tremaining: 16.4s\n",
            "430:\ttotal: 12.5s\tremaining: 16.4s\n",
            "431:\ttotal: 12.5s\tremaining: 16.4s\n",
            "432:\ttotal: 12.5s\tremaining: 16.3s\n",
            "433:\ttotal: 12.5s\tremaining: 16.3s\n",
            "434:\ttotal: 12.6s\tremaining: 16.3s\n",
            "435:\ttotal: 12.6s\tremaining: 16.3s\n",
            "436:\ttotal: 12.6s\tremaining: 16.2s\n",
            "437:\ttotal: 12.7s\tremaining: 16.2s\n",
            "438:\ttotal: 12.7s\tremaining: 16.2s\n",
            "439:\ttotal: 12.7s\tremaining: 16.1s\n",
            "440:\ttotal: 12.7s\tremaining: 16.1s\n",
            "441:\ttotal: 12.8s\tremaining: 16.1s\n",
            "442:\ttotal: 12.8s\tremaining: 16s\n",
            "443:\ttotal: 12.8s\tremaining: 16s\n",
            "444:\ttotal: 12.9s\tremaining: 16s\n",
            "445:\ttotal: 12.9s\tremaining: 15.9s\n",
            "446:\ttotal: 12.9s\tremaining: 15.9s\n",
            "447:\ttotal: 12.9s\tremaining: 15.9s\n",
            "448:\ttotal: 13s\tremaining: 15.8s\n",
            "449:\ttotal: 13s\tremaining: 15.8s\n",
            "450:\ttotal: 13s\tremaining: 15.8s\n",
            "451:\ttotal: 13s\tremaining: 15.8s\n",
            "452:\ttotal: 13.1s\tremaining: 15.7s\n",
            "453:\ttotal: 13.1s\tremaining: 15.7s\n",
            "454:\ttotal: 13.1s\tremaining: 15.7s\n",
            "455:\ttotal: 13.2s\tremaining: 15.6s\n",
            "456:\ttotal: 13.2s\tremaining: 15.6s\n",
            "457:\ttotal: 13.2s\tremaining: 15.6s\n",
            "458:\ttotal: 13.2s\tremaining: 15.5s\n",
            "459:\ttotal: 13.3s\tremaining: 15.5s\n",
            "460:\ttotal: 13.3s\tremaining: 15.5s\n",
            "461:\ttotal: 13.3s\tremaining: 15.5s\n",
            "462:\ttotal: 13.3s\tremaining: 15.4s\n",
            "463:\ttotal: 13.4s\tremaining: 15.4s\n",
            "464:\ttotal: 13.4s\tremaining: 15.4s\n",
            "465:\ttotal: 13.4s\tremaining: 15.3s\n",
            "466:\ttotal: 13.5s\tremaining: 15.3s\n",
            "467:\ttotal: 13.5s\tremaining: 15.3s\n",
            "468:\ttotal: 13.5s\tremaining: 15.2s\n",
            "469:\ttotal: 13.5s\tremaining: 15.2s\n",
            "470:\ttotal: 13.6s\tremaining: 15.2s\n",
            "471:\ttotal: 13.6s\tremaining: 15.1s\n",
            "472:\ttotal: 13.6s\tremaining: 15.1s\n",
            "473:\ttotal: 13.7s\tremaining: 15.1s\n",
            "474:\ttotal: 13.7s\tremaining: 15.1s\n",
            "475:\ttotal: 13.7s\tremaining: 15s\n",
            "476:\ttotal: 13.7s\tremaining: 15s\n",
            "477:\ttotal: 13.8s\tremaining: 15s\n",
            "478:\ttotal: 13.8s\tremaining: 15s\n",
            "479:\ttotal: 13.8s\tremaining: 14.9s\n",
            "480:\ttotal: 13.9s\tremaining: 14.9s\n",
            "481:\ttotal: 13.9s\tremaining: 14.9s\n",
            "482:\ttotal: 13.9s\tremaining: 14.9s\n",
            "483:\ttotal: 14s\tremaining: 14.8s\n",
            "484:\ttotal: 14s\tremaining: 14.8s\n",
            "485:\ttotal: 14s\tremaining: 14.8s\n",
            "486:\ttotal: 14s\tremaining: 14.7s\n",
            "487:\ttotal: 14.1s\tremaining: 14.7s\n",
            "488:\ttotal: 14.1s\tremaining: 14.7s\n",
            "489:\ttotal: 14.1s\tremaining: 14.7s\n",
            "490:\ttotal: 14.2s\tremaining: 14.6s\n",
            "491:\ttotal: 14.2s\tremaining: 14.6s\n",
            "492:\ttotal: 14.2s\tremaining: 14.6s\n",
            "493:\ttotal: 14.3s\tremaining: 14.6s\n",
            "494:\ttotal: 14.3s\tremaining: 14.5s\n",
            "495:\ttotal: 14.3s\tremaining: 14.5s\n",
            "496:\ttotal: 14.4s\tremaining: 14.5s\n",
            "497:\ttotal: 14.4s\tremaining: 14.5s\n",
            "498:\ttotal: 14.4s\tremaining: 14.4s\n",
            "499:\ttotal: 14.5s\tremaining: 14.4s\n",
            "500:\ttotal: 14.5s\tremaining: 14.4s\n",
            "501:\ttotal: 14.5s\tremaining: 14.4s\n",
            "502:\ttotal: 14.6s\tremaining: 14.3s\n",
            "503:\ttotal: 14.6s\tremaining: 14.3s\n",
            "504:\ttotal: 14.6s\tremaining: 14.3s\n",
            "505:\ttotal: 14.7s\tremaining: 14.3s\n",
            "506:\ttotal: 14.7s\tremaining: 14.2s\n",
            "507:\ttotal: 14.7s\tremaining: 14.2s\n",
            "508:\ttotal: 14.7s\tremaining: 14.2s\n",
            "509:\ttotal: 14.8s\tremaining: 14.1s\n",
            "510:\ttotal: 14.8s\tremaining: 14.1s\n",
            "511:\ttotal: 14.8s\tremaining: 14.1s\n",
            "512:\ttotal: 14.9s\tremaining: 14.1s\n",
            "513:\ttotal: 14.9s\tremaining: 14s\n",
            "514:\ttotal: 14.9s\tremaining: 14s\n",
            "515:\ttotal: 15s\tremaining: 14s\n",
            "516:\ttotal: 15s\tremaining: 13.9s\n",
            "517:\ttotal: 15s\tremaining: 13.9s\n",
            "518:\ttotal: 15.1s\tremaining: 13.9s\n",
            "519:\ttotal: 15.1s\tremaining: 13.9s\n",
            "520:\ttotal: 15.1s\tremaining: 13.8s\n",
            "521:\ttotal: 15.2s\tremaining: 13.8s\n",
            "522:\ttotal: 15.2s\tremaining: 13.8s\n",
            "523:\ttotal: 15.2s\tremaining: 13.8s\n",
            "524:\ttotal: 15.2s\tremaining: 13.7s\n",
            "525:\ttotal: 15.3s\tremaining: 13.7s\n",
            "526:\ttotal: 15.3s\tremaining: 13.7s\n",
            "527:\ttotal: 15.3s\tremaining: 13.6s\n",
            "528:\ttotal: 15.4s\tremaining: 13.6s\n",
            "529:\ttotal: 15.4s\tremaining: 13.6s\n",
            "530:\ttotal: 15.4s\tremaining: 13.6s\n",
            "531:\ttotal: 15.5s\tremaining: 13.5s\n",
            "532:\ttotal: 15.5s\tremaining: 13.5s\n",
            "533:\ttotal: 15.5s\tremaining: 13.5s\n",
            "534:\ttotal: 15.6s\tremaining: 13.5s\n",
            "535:\ttotal: 15.6s\tremaining: 13.4s\n",
            "536:\ttotal: 15.6s\tremaining: 13.4s\n",
            "537:\ttotal: 15.6s\tremaining: 13.4s\n",
            "538:\ttotal: 15.7s\tremaining: 13.3s\n",
            "539:\ttotal: 15.7s\tremaining: 13.3s\n",
            "540:\ttotal: 15.7s\tremaining: 13.3s\n",
            "541:\ttotal: 15.8s\tremaining: 13.3s\n",
            "542:\ttotal: 15.8s\tremaining: 13.2s\n",
            "543:\ttotal: 15.8s\tremaining: 13.2s\n",
            "544:\ttotal: 15.9s\tremaining: 13.2s\n",
            "545:\ttotal: 15.9s\tremaining: 13.2s\n",
            "546:\ttotal: 15.9s\tremaining: 13.1s\n",
            "547:\ttotal: 16s\tremaining: 13.1s\n",
            "548:\ttotal: 16s\tremaining: 13.1s\n",
            "549:\ttotal: 16s\tremaining: 13.1s\n",
            "550:\ttotal: 16.1s\tremaining: 13s\n",
            "551:\ttotal: 16.1s\tremaining: 13s\n",
            "552:\ttotal: 16.1s\tremaining: 13s\n",
            "553:\ttotal: 16.2s\tremaining: 13s\n",
            "554:\ttotal: 16.2s\tremaining: 12.9s\n",
            "555:\ttotal: 16.2s\tremaining: 12.9s\n",
            "556:\ttotal: 16.3s\tremaining: 12.9s\n",
            "557:\ttotal: 16.3s\tremaining: 12.9s\n",
            "558:\ttotal: 16.3s\tremaining: 12.8s\n",
            "559:\ttotal: 16.4s\tremaining: 12.8s\n",
            "560:\ttotal: 16.4s\tremaining: 12.8s\n",
            "561:\ttotal: 16.4s\tremaining: 12.7s\n",
            "562:\ttotal: 16.4s\tremaining: 12.7s\n",
            "563:\ttotal: 16.5s\tremaining: 12.7s\n",
            "564:\ttotal: 16.5s\tremaining: 12.7s\n",
            "565:\ttotal: 16.5s\tremaining: 12.6s\n",
            "566:\ttotal: 16.6s\tremaining: 12.6s\n",
            "567:\ttotal: 16.6s\tremaining: 12.6s\n",
            "568:\ttotal: 16.6s\tremaining: 12.5s\n",
            "569:\ttotal: 16.7s\tremaining: 12.5s\n",
            "570:\ttotal: 16.7s\tremaining: 12.5s\n",
            "571:\ttotal: 16.7s\tremaining: 12.5s\n",
            "572:\ttotal: 16.8s\tremaining: 12.4s\n",
            "573:\ttotal: 16.8s\tremaining: 12.4s\n",
            "574:\ttotal: 16.8s\tremaining: 12.4s\n",
            "575:\ttotal: 16.8s\tremaining: 12.3s\n",
            "576:\ttotal: 16.9s\tremaining: 12.3s\n",
            "577:\ttotal: 16.9s\tremaining: 12.3s\n",
            "578:\ttotal: 16.9s\tremaining: 12.3s\n",
            "579:\ttotal: 17s\tremaining: 12.2s\n",
            "580:\ttotal: 17s\tremaining: 12.2s\n",
            "581:\ttotal: 17s\tremaining: 12.2s\n",
            "582:\ttotal: 17.1s\tremaining: 12.1s\n",
            "583:\ttotal: 17.1s\tremaining: 12.1s\n",
            "584:\ttotal: 17.1s\tremaining: 12.1s\n",
            "585:\ttotal: 17.2s\tremaining: 12.1s\n",
            "586:\ttotal: 17.2s\tremaining: 12s\n",
            "587:\ttotal: 17.2s\tremaining: 12s\n",
            "588:\ttotal: 17.2s\tremaining: 12s\n",
            "589:\ttotal: 17.3s\tremaining: 11.9s\n",
            "590:\ttotal: 17.3s\tremaining: 11.9s\n",
            "591:\ttotal: 17.3s\tremaining: 11.9s\n",
            "592:\ttotal: 17.4s\tremaining: 11.9s\n",
            "593:\ttotal: 17.4s\tremaining: 11.8s\n",
            "594:\ttotal: 17.4s\tremaining: 11.8s\n",
            "595:\ttotal: 17.5s\tremaining: 11.8s\n",
            "596:\ttotal: 17.5s\tremaining: 11.8s\n",
            "597:\ttotal: 17.5s\tremaining: 11.7s\n",
            "598:\ttotal: 17.6s\tremaining: 11.7s\n",
            "599:\ttotal: 17.6s\tremaining: 11.7s\n",
            "600:\ttotal: 17.6s\tremaining: 11.6s\n",
            "601:\ttotal: 17.7s\tremaining: 11.6s\n",
            "602:\ttotal: 17.7s\tremaining: 11.6s\n",
            "603:\ttotal: 17.7s\tremaining: 11.6s\n",
            "604:\ttotal: 17.7s\tremaining: 11.5s\n",
            "605:\ttotal: 17.8s\tremaining: 11.5s\n",
            "606:\ttotal: 17.8s\tremaining: 11.5s\n",
            "607:\ttotal: 17.8s\tremaining: 11.4s\n",
            "608:\ttotal: 17.9s\tremaining: 11.4s\n",
            "609:\ttotal: 17.9s\tremaining: 11.4s\n",
            "610:\ttotal: 17.9s\tremaining: 11.4s\n",
            "611:\ttotal: 18s\tremaining: 11.3s\n",
            "612:\ttotal: 18s\tremaining: 11.3s\n",
            "613:\ttotal: 18s\tremaining: 11.3s\n",
            "614:\ttotal: 18.1s\tremaining: 11.2s\n",
            "615:\ttotal: 18.1s\tremaining: 11.2s\n",
            "616:\ttotal: 18.1s\tremaining: 11.2s\n",
            "617:\ttotal: 18.1s\tremaining: 11.2s\n",
            "618:\ttotal: 18.2s\tremaining: 11.1s\n",
            "619:\ttotal: 18.2s\tremaining: 11.1s\n",
            "620:\ttotal: 18.2s\tremaining: 11.1s\n",
            "621:\ttotal: 18.3s\tremaining: 11s\n",
            "622:\ttotal: 18.3s\tremaining: 11s\n",
            "623:\ttotal: 18.3s\tremaining: 11s\n",
            "624:\ttotal: 18.4s\tremaining: 11s\n",
            "625:\ttotal: 18.4s\tremaining: 10.9s\n",
            "626:\ttotal: 18.4s\tremaining: 10.9s\n",
            "627:\ttotal: 18.4s\tremaining: 10.9s\n",
            "628:\ttotal: 18.5s\tremaining: 10.8s\n",
            "629:\ttotal: 18.5s\tremaining: 10.8s\n",
            "630:\ttotal: 18.5s\tremaining: 10.8s\n",
            "631:\ttotal: 18.6s\tremaining: 10.8s\n",
            "632:\ttotal: 18.6s\tremaining: 10.7s\n",
            "633:\ttotal: 18.6s\tremaining: 10.7s\n",
            "634:\ttotal: 18.7s\tremaining: 10.7s\n",
            "635:\ttotal: 18.7s\tremaining: 10.6s\n",
            "636:\ttotal: 18.7s\tremaining: 10.6s\n",
            "637:\ttotal: 18.8s\tremaining: 10.6s\n",
            "638:\ttotal: 18.8s\tremaining: 10.6s\n",
            "639:\ttotal: 18.8s\tremaining: 10.5s\n",
            "640:\ttotal: 18.9s\tremaining: 10.5s\n",
            "641:\ttotal: 18.9s\tremaining: 10.5s\n",
            "642:\ttotal: 18.9s\tremaining: 10.4s\n",
            "643:\ttotal: 19s\tremaining: 10.4s\n",
            "644:\ttotal: 19s\tremaining: 10.4s\n",
            "645:\ttotal: 19s\tremaining: 10.4s\n",
            "646:\ttotal: 19s\tremaining: 10.3s\n",
            "647:\ttotal: 19.1s\tremaining: 10.3s\n",
            "648:\ttotal: 19.1s\tremaining: 10.3s\n",
            "649:\ttotal: 19.1s\tremaining: 10.2s\n",
            "650:\ttotal: 19.2s\tremaining: 10.2s\n",
            "651:\ttotal: 19.2s\tremaining: 10.2s\n",
            "652:\ttotal: 19.2s\tremaining: 10.2s\n",
            "653:\ttotal: 19.3s\tremaining: 10.1s\n",
            "654:\ttotal: 19.3s\tremaining: 10.1s\n",
            "655:\ttotal: 19.3s\tremaining: 10.1s\n",
            "656:\ttotal: 19.4s\tremaining: 10s\n",
            "657:\ttotal: 19.4s\tremaining: 10s\n",
            "658:\ttotal: 19.4s\tremaining: 9.99s\n",
            "659:\ttotal: 19.4s\tremaining: 9.96s\n",
            "660:\ttotal: 19.5s\tremaining: 9.93s\n",
            "661:\ttotal: 19.5s\tremaining: 9.89s\n",
            "662:\ttotal: 19.5s\tremaining: 9.87s\n",
            "663:\ttotal: 19.6s\tremaining: 9.84s\n",
            "664:\ttotal: 19.6s\tremaining: 9.81s\n",
            "665:\ttotal: 19.6s\tremaining: 9.78s\n",
            "666:\ttotal: 19.6s\tremaining: 9.75s\n",
            "667:\ttotal: 19.7s\tremaining: 9.72s\n",
            "668:\ttotal: 19.7s\tremaining: 9.69s\n",
            "669:\ttotal: 19.7s\tremaining: 9.66s\n",
            "670:\ttotal: 19.8s\tremaining: 9.63s\n",
            "671:\ttotal: 19.8s\tremaining: 9.61s\n",
            "672:\ttotal: 19.8s\tremaining: 9.57s\n",
            "673:\ttotal: 19.9s\tremaining: 9.54s\n",
            "674:\ttotal: 19.9s\tremaining: 9.52s\n",
            "675:\ttotal: 19.9s\tremaining: 9.49s\n",
            "676:\ttotal: 20s\tremaining: 9.46s\n",
            "677:\ttotal: 20s\tremaining: 9.43s\n",
            "678:\ttotal: 20s\tremaining: 9.4s\n",
            "679:\ttotal: 20s\tremaining: 9.37s\n",
            "680:\ttotal: 20.1s\tremaining: 9.34s\n",
            "681:\ttotal: 20.1s\tremaining: 9.31s\n",
            "682:\ttotal: 20.1s\tremaining: 9.28s\n",
            "683:\ttotal: 20.2s\tremaining: 9.26s\n",
            "684:\ttotal: 20.2s\tremaining: 9.23s\n",
            "685:\ttotal: 20.2s\tremaining: 9.2s\n",
            "686:\ttotal: 20.3s\tremaining: 9.17s\n",
            "687:\ttotal: 20.3s\tremaining: 9.14s\n",
            "688:\ttotal: 20.3s\tremaining: 9.11s\n",
            "689:\ttotal: 20.3s\tremaining: 9.08s\n",
            "690:\ttotal: 20.4s\tremaining: 9.05s\n",
            "691:\ttotal: 20.4s\tremaining: 9.02s\n",
            "692:\ttotal: 20.4s\tremaining: 8.99s\n",
            "693:\ttotal: 20.5s\tremaining: 8.96s\n",
            "694:\ttotal: 20.5s\tremaining: 8.93s\n",
            "695:\ttotal: 20.5s\tremaining: 8.9s\n",
            "696:\ttotal: 20.5s\tremaining: 8.87s\n",
            "697:\ttotal: 20.6s\tremaining: 8.84s\n",
            "698:\ttotal: 20.6s\tremaining: 8.81s\n",
            "699:\ttotal: 20.6s\tremaining: 8.78s\n",
            "700:\ttotal: 20.7s\tremaining: 8.75s\n",
            "701:\ttotal: 20.7s\tremaining: 8.72s\n",
            "702:\ttotal: 20.7s\tremaining: 8.7s\n",
            "703:\ttotal: 20.8s\tremaining: 8.67s\n",
            "704:\ttotal: 20.8s\tremaining: 8.64s\n",
            "705:\ttotal: 20.8s\tremaining: 8.61s\n",
            "706:\ttotal: 20.8s\tremaining: 8.58s\n",
            "707:\ttotal: 20.9s\tremaining: 8.55s\n",
            "708:\ttotal: 20.9s\tremaining: 8.52s\n",
            "709:\ttotal: 20.9s\tremaining: 8.49s\n",
            "710:\ttotal: 21s\tremaining: 8.46s\n",
            "711:\ttotal: 21s\tremaining: 8.44s\n",
            "712:\ttotal: 21s\tremaining: 8.41s\n",
            "713:\ttotal: 21.1s\tremaining: 8.38s\n",
            "714:\ttotal: 21.1s\tremaining: 8.35s\n",
            "715:\ttotal: 21.1s\tremaining: 8.32s\n",
            "716:\ttotal: 21.2s\tremaining: 8.29s\n",
            "717:\ttotal: 21.2s\tremaining: 8.26s\n",
            "718:\ttotal: 21.2s\tremaining: 8.23s\n",
            "719:\ttotal: 21.2s\tremaining: 8.2s\n",
            "720:\ttotal: 21.3s\tremaining: 8.17s\n",
            "721:\ttotal: 21.3s\tremaining: 8.14s\n",
            "722:\ttotal: 21.3s\tremaining: 8.12s\n",
            "723:\ttotal: 21.4s\tremaining: 8.09s\n",
            "724:\ttotal: 21.4s\tremaining: 8.05s\n",
            "725:\ttotal: 21.4s\tremaining: 8.03s\n",
            "726:\ttotal: 21.5s\tremaining: 8s\n",
            "727:\ttotal: 21.5s\tremaining: 7.97s\n",
            "728:\ttotal: 21.5s\tremaining: 7.94s\n",
            "729:\ttotal: 21.5s\tremaining: 7.91s\n",
            "730:\ttotal: 21.6s\tremaining: 7.88s\n",
            "731:\ttotal: 21.6s\tremaining: 7.85s\n",
            "732:\ttotal: 21.6s\tremaining: 7.83s\n",
            "733:\ttotal: 21.7s\tremaining: 7.8s\n",
            "734:\ttotal: 21.7s\tremaining: 7.77s\n",
            "735:\ttotal: 21.7s\tremaining: 7.74s\n",
            "736:\ttotal: 21.8s\tremaining: 7.71s\n",
            "737:\ttotal: 21.8s\tremaining: 7.68s\n",
            "738:\ttotal: 21.8s\tremaining: 7.65s\n",
            "739:\ttotal: 21.9s\tremaining: 7.63s\n",
            "740:\ttotal: 21.9s\tremaining: 7.6s\n",
            "741:\ttotal: 21.9s\tremaining: 7.57s\n",
            "742:\ttotal: 22s\tremaining: 7.54s\n",
            "743:\ttotal: 22s\tremaining: 7.51s\n",
            "744:\ttotal: 22s\tremaining: 7.48s\n",
            "745:\ttotal: 22.1s\tremaining: 7.45s\n",
            "746:\ttotal: 22.1s\tremaining: 7.42s\n",
            "747:\ttotal: 22.1s\tremaining: 7.39s\n",
            "748:\ttotal: 22.1s\tremaining: 7.36s\n",
            "749:\ttotal: 22.2s\tremaining: 7.33s\n",
            "750:\ttotal: 22.2s\tremaining: 7.3s\n",
            "751:\ttotal: 22.2s\tremaining: 7.27s\n",
            "752:\ttotal: 22.3s\tremaining: 7.24s\n",
            "753:\ttotal: 22.3s\tremaining: 7.22s\n",
            "754:\ttotal: 22.3s\tremaining: 7.19s\n",
            "755:\ttotal: 22.4s\tremaining: 7.16s\n",
            "756:\ttotal: 22.4s\tremaining: 7.13s\n",
            "757:\ttotal: 22.4s\tremaining: 7.1s\n",
            "758:\ttotal: 22.4s\tremaining: 7.07s\n",
            "759:\ttotal: 22.5s\tremaining: 7.04s\n",
            "760:\ttotal: 22.5s\tremaining: 7.01s\n",
            "761:\ttotal: 22.5s\tremaining: 6.98s\n",
            "762:\ttotal: 22.6s\tremaining: 6.95s\n",
            "763:\ttotal: 22.6s\tremaining: 6.92s\n",
            "764:\ttotal: 22.6s\tremaining: 6.89s\n",
            "765:\ttotal: 22.7s\tremaining: 6.86s\n",
            "766:\ttotal: 22.7s\tremaining: 6.83s\n",
            "767:\ttotal: 22.7s\tremaining: 6.8s\n",
            "768:\ttotal: 22.7s\tremaining: 6.77s\n",
            "769:\ttotal: 22.8s\tremaining: 6.74s\n",
            "770:\ttotal: 22.8s\tremaining: 6.71s\n",
            "771:\ttotal: 22.8s\tremaining: 6.68s\n",
            "772:\ttotal: 22.9s\tremaining: 6.65s\n",
            "773:\ttotal: 22.9s\tremaining: 6.62s\n",
            "774:\ttotal: 22.9s\tremaining: 6.6s\n",
            "775:\ttotal: 23s\tremaining: 6.57s\n",
            "776:\ttotal: 23s\tremaining: 6.54s\n",
            "777:\ttotal: 23s\tremaining: 6.51s\n",
            "778:\ttotal: 23.1s\tremaining: 6.48s\n",
            "779:\ttotal: 23.1s\tremaining: 6.45s\n",
            "780:\ttotal: 23.1s\tremaining: 6.42s\n",
            "781:\ttotal: 23.2s\tremaining: 6.39s\n",
            "782:\ttotal: 23.2s\tremaining: 6.37s\n",
            "783:\ttotal: 23.2s\tremaining: 6.33s\n",
            "784:\ttotal: 23.2s\tremaining: 6.3s\n",
            "785:\ttotal: 23.3s\tremaining: 6.28s\n",
            "786:\ttotal: 23.3s\tremaining: 6.25s\n",
            "787:\ttotal: 23.3s\tremaining: 6.22s\n",
            "788:\ttotal: 23.4s\tremaining: 6.19s\n",
            "789:\ttotal: 23.4s\tremaining: 6.16s\n",
            "790:\ttotal: 23.4s\tremaining: 6.13s\n",
            "791:\ttotal: 23.5s\tremaining: 6.1s\n",
            "792:\ttotal: 23.5s\tremaining: 6.07s\n",
            "793:\ttotal: 23.5s\tremaining: 6.04s\n",
            "794:\ttotal: 23.5s\tremaining: 6.01s\n",
            "795:\ttotal: 23.6s\tremaining: 5.98s\n",
            "796:\ttotal: 23.6s\tremaining: 5.95s\n",
            "797:\ttotal: 23.6s\tremaining: 5.92s\n",
            "798:\ttotal: 23.7s\tremaining: 5.89s\n",
            "799:\ttotal: 23.7s\tremaining: 5.86s\n",
            "800:\ttotal: 23.7s\tremaining: 5.83s\n",
            "801:\ttotal: 23.8s\tremaining: 5.8s\n",
            "802:\ttotal: 23.8s\tremaining: 5.78s\n",
            "803:\ttotal: 23.8s\tremaining: 5.75s\n",
            "804:\ttotal: 23.9s\tremaining: 5.72s\n",
            "805:\ttotal: 23.9s\tremaining: 5.69s\n",
            "806:\ttotal: 23.9s\tremaining: 5.66s\n",
            "807:\ttotal: 23.9s\tremaining: 5.63s\n",
            "808:\ttotal: 24s\tremaining: 5.6s\n",
            "809:\ttotal: 24s\tremaining: 5.57s\n",
            "810:\ttotal: 24s\tremaining: 5.54s\n",
            "811:\ttotal: 24.1s\tremaining: 5.51s\n",
            "812:\ttotal: 24.1s\tremaining: 5.48s\n",
            "813:\ttotal: 24.1s\tremaining: 5.45s\n",
            "814:\ttotal: 24.1s\tremaining: 5.42s\n",
            "815:\ttotal: 24.2s\tremaining: 5.39s\n",
            "816:\ttotal: 24.2s\tremaining: 5.36s\n",
            "817:\ttotal: 24.2s\tremaining: 5.33s\n",
            "818:\ttotal: 24.3s\tremaining: 5.3s\n",
            "819:\ttotal: 24.3s\tremaining: 5.27s\n",
            "820:\ttotal: 24.3s\tremaining: 5.24s\n",
            "821:\ttotal: 24.4s\tremaining: 5.21s\n",
            "822:\ttotal: 24.4s\tremaining: 5.18s\n",
            "823:\ttotal: 24.4s\tremaining: 5.16s\n",
            "824:\ttotal: 24.4s\tremaining: 5.13s\n",
            "825:\ttotal: 24.5s\tremaining: 5.1s\n",
            "826:\ttotal: 24.5s\tremaining: 5.07s\n",
            "827:\ttotal: 24.5s\tremaining: 5.04s\n",
            "828:\ttotal: 24.6s\tremaining: 5.01s\n",
            "829:\ttotal: 24.6s\tremaining: 4.98s\n",
            "830:\ttotal: 24.6s\tremaining: 4.95s\n",
            "831:\ttotal: 24.7s\tremaining: 4.92s\n",
            "832:\ttotal: 24.7s\tremaining: 4.89s\n",
            "833:\ttotal: 24.7s\tremaining: 4.86s\n",
            "834:\ttotal: 24.7s\tremaining: 4.83s\n",
            "835:\ttotal: 24.8s\tremaining: 4.8s\n",
            "836:\ttotal: 24.8s\tremaining: 4.77s\n",
            "837:\ttotal: 24.8s\tremaining: 4.74s\n",
            "838:\ttotal: 24.9s\tremaining: 4.71s\n",
            "839:\ttotal: 24.9s\tremaining: 4.68s\n",
            "840:\ttotal: 24.9s\tremaining: 4.65s\n",
            "841:\ttotal: 24.9s\tremaining: 4.62s\n",
            "842:\ttotal: 25s\tremaining: 4.59s\n",
            "843:\ttotal: 25s\tremaining: 4.56s\n",
            "844:\ttotal: 25.1s\tremaining: 4.54s\n",
            "845:\ttotal: 25.1s\tremaining: 4.51s\n",
            "846:\ttotal: 25.1s\tremaining: 4.48s\n",
            "847:\ttotal: 25.1s\tremaining: 4.45s\n",
            "848:\ttotal: 25.2s\tremaining: 4.42s\n",
            "849:\ttotal: 25.2s\tremaining: 4.39s\n",
            "850:\ttotal: 25.2s\tremaining: 4.36s\n",
            "851:\ttotal: 25.3s\tremaining: 4.33s\n",
            "852:\ttotal: 25.3s\tremaining: 4.3s\n",
            "853:\ttotal: 25.3s\tremaining: 4.27s\n",
            "854:\ttotal: 25.4s\tremaining: 4.24s\n",
            "855:\ttotal: 25.4s\tremaining: 4.21s\n",
            "856:\ttotal: 25.4s\tremaining: 4.18s\n",
            "857:\ttotal: 25.4s\tremaining: 4.15s\n",
            "858:\ttotal: 25.5s\tremaining: 4.12s\n",
            "859:\ttotal: 25.5s\tremaining: 4.09s\n",
            "860:\ttotal: 25.5s\tremaining: 4.06s\n",
            "861:\ttotal: 25.6s\tremaining: 4.03s\n",
            "862:\ttotal: 25.6s\tremaining: 4s\n",
            "863:\ttotal: 25.6s\tremaining: 3.97s\n",
            "864:\ttotal: 25.6s\tremaining: 3.94s\n",
            "865:\ttotal: 25.7s\tremaining: 3.92s\n",
            "866:\ttotal: 25.7s\tremaining: 3.88s\n",
            "867:\ttotal: 25.7s\tremaining: 3.85s\n",
            "868:\ttotal: 25.8s\tremaining: 3.83s\n",
            "869:\ttotal: 25.8s\tremaining: 3.79s\n",
            "870:\ttotal: 25.8s\tremaining: 3.77s\n",
            "871:\ttotal: 25.9s\tremaining: 3.74s\n",
            "872:\ttotal: 25.9s\tremaining: 3.71s\n",
            "873:\ttotal: 25.9s\tremaining: 3.68s\n",
            "874:\ttotal: 25.9s\tremaining: 3.65s\n",
            "875:\ttotal: 26s\tremaining: 3.62s\n",
            "876:\ttotal: 26s\tremaining: 3.59s\n",
            "877:\ttotal: 26s\tremaining: 3.56s\n",
            "878:\ttotal: 26.1s\tremaining: 3.53s\n",
            "879:\ttotal: 26.1s\tremaining: 3.5s\n",
            "880:\ttotal: 26.1s\tremaining: 3.47s\n",
            "881:\ttotal: 26.2s\tremaining: 3.44s\n",
            "882:\ttotal: 26.2s\tremaining: 3.41s\n",
            "883:\ttotal: 26.2s\tremaining: 3.38s\n",
            "884:\ttotal: 26.2s\tremaining: 3.35s\n",
            "885:\ttotal: 26.3s\tremaining: 3.32s\n",
            "886:\ttotal: 26.3s\tremaining: 3.29s\n",
            "887:\ttotal: 26.3s\tremaining: 3.26s\n",
            "888:\ttotal: 26.4s\tremaining: 3.23s\n",
            "889:\ttotal: 26.4s\tremaining: 3.2s\n",
            "890:\ttotal: 26.4s\tremaining: 3.17s\n",
            "891:\ttotal: 26.4s\tremaining: 3.14s\n",
            "892:\ttotal: 26.5s\tremaining: 3.11s\n",
            "893:\ttotal: 26.5s\tremaining: 3.08s\n",
            "894:\ttotal: 26.5s\tremaining: 3.05s\n",
            "895:\ttotal: 26.6s\tremaining: 3.02s\n",
            "896:\ttotal: 26.6s\tremaining: 2.99s\n",
            "897:\ttotal: 26.6s\tremaining: 2.96s\n",
            "898:\ttotal: 26.6s\tremaining: 2.93s\n",
            "899:\ttotal: 26.7s\tremaining: 2.9s\n",
            "900:\ttotal: 26.7s\tremaining: 2.87s\n",
            "901:\ttotal: 26.7s\tremaining: 2.84s\n",
            "902:\ttotal: 26.7s\tremaining: 2.81s\n",
            "903:\ttotal: 26.8s\tremaining: 2.78s\n",
            "904:\ttotal: 26.8s\tremaining: 2.75s\n",
            "905:\ttotal: 26.8s\tremaining: 2.72s\n",
            "906:\ttotal: 26.9s\tremaining: 2.69s\n",
            "907:\ttotal: 26.9s\tremaining: 2.66s\n",
            "908:\ttotal: 26.9s\tremaining: 2.63s\n",
            "909:\ttotal: 26.9s\tremaining: 2.6s\n",
            "910:\ttotal: 27s\tremaining: 2.57s\n",
            "911:\ttotal: 27s\tremaining: 2.54s\n",
            "912:\ttotal: 27s\tremaining: 2.52s\n",
            "913:\ttotal: 27s\tremaining: 2.48s\n",
            "914:\ttotal: 27.1s\tremaining: 2.46s\n",
            "915:\ttotal: 27.1s\tremaining: 2.42s\n",
            "916:\ttotal: 27.1s\tremaining: 2.4s\n",
            "917:\ttotal: 27.1s\tremaining: 2.37s\n",
            "918:\ttotal: 27.2s\tremaining: 2.33s\n",
            "919:\ttotal: 27.2s\tremaining: 2.31s\n",
            "920:\ttotal: 27.2s\tremaining: 2.28s\n",
            "921:\ttotal: 27.2s\tremaining: 2.25s\n",
            "922:\ttotal: 27.3s\tremaining: 2.22s\n",
            "923:\ttotal: 27.3s\tremaining: 2.19s\n",
            "924:\ttotal: 27.3s\tremaining: 2.16s\n",
            "925:\ttotal: 27.4s\tremaining: 2.13s\n",
            "926:\ttotal: 27.4s\tremaining: 2.1s\n",
            "927:\ttotal: 27.4s\tremaining: 2.07s\n",
            "928:\ttotal: 27.4s\tremaining: 2.04s\n",
            "929:\ttotal: 27.5s\tremaining: 2.01s\n",
            "930:\ttotal: 27.5s\tremaining: 1.98s\n",
            "931:\ttotal: 27.5s\tremaining: 1.95s\n",
            "932:\ttotal: 27.6s\tremaining: 1.92s\n",
            "933:\ttotal: 27.6s\tremaining: 1.89s\n",
            "934:\ttotal: 27.6s\tremaining: 1.86s\n",
            "935:\ttotal: 27.7s\tremaining: 1.83s\n",
            "936:\ttotal: 27.7s\tremaining: 1.8s\n",
            "937:\ttotal: 27.7s\tremaining: 1.77s\n",
            "938:\ttotal: 27.7s\tremaining: 1.74s\n",
            "939:\ttotal: 27.8s\tremaining: 1.71s\n",
            "940:\ttotal: 27.8s\tremaining: 1.68s\n",
            "941:\ttotal: 27.8s\tremaining: 1.65s\n",
            "942:\ttotal: 27.9s\tremaining: 1.62s\n",
            "943:\ttotal: 27.9s\tremaining: 1.59s\n",
            "944:\ttotal: 27.9s\tremaining: 1.56s\n",
            "945:\ttotal: 27.9s\tremaining: 1.54s\n",
            "946:\ttotal: 28s\tremaining: 1.51s\n",
            "947:\ttotal: 28s\tremaining: 1.48s\n",
            "948:\ttotal: 28s\tremaining: 1.45s\n",
            "949:\ttotal: 28.1s\tremaining: 1.42s\n",
            "950:\ttotal: 28.1s\tremaining: 1.39s\n",
            "951:\ttotal: 28.1s\tremaining: 1.36s\n",
            "952:\ttotal: 28.2s\tremaining: 1.33s\n",
            "953:\ttotal: 28.2s\tremaining: 1.3s\n",
            "954:\ttotal: 28.2s\tremaining: 1.27s\n",
            "955:\ttotal: 28.3s\tremaining: 1.24s\n",
            "956:\ttotal: 28.3s\tremaining: 1.21s\n",
            "957:\ttotal: 28.3s\tremaining: 1.18s\n",
            "958:\ttotal: 28.3s\tremaining: 1.15s\n",
            "959:\ttotal: 28.4s\tremaining: 1.12s\n",
            "960:\ttotal: 28.4s\tremaining: 1.09s\n",
            "961:\ttotal: 28.4s\tremaining: 1.06s\n",
            "962:\ttotal: 28.5s\tremaining: 1.03s\n",
            "963:\ttotal: 28.5s\tremaining: 1s\n",
            "964:\ttotal: 28.5s\tremaining: 975ms\n",
            "965:\ttotal: 28.6s\tremaining: 946ms\n",
            "966:\ttotal: 28.6s\tremaining: 916ms\n",
            "967:\ttotal: 28.6s\tremaining: 887ms\n",
            "968:\ttotal: 28.6s\tremaining: 857ms\n",
            "969:\ttotal: 28.7s\tremaining: 828ms\n",
            "970:\ttotal: 28.7s\tremaining: 798ms\n",
            "971:\ttotal: 28.7s\tremaining: 769ms\n",
            "972:\ttotal: 28.8s\tremaining: 739ms\n",
            "973:\ttotal: 28.8s\tremaining: 709ms\n",
            "974:\ttotal: 28.8s\tremaining: 680ms\n",
            "975:\ttotal: 28.9s\tremaining: 650ms\n",
            "976:\ttotal: 28.9s\tremaining: 621ms\n",
            "977:\ttotal: 28.9s\tremaining: 591ms\n",
            "978:\ttotal: 28.9s\tremaining: 562ms\n",
            "979:\ttotal: 29s\tremaining: 532ms\n",
            "980:\ttotal: 29s\tremaining: 503ms\n",
            "981:\ttotal: 29s\tremaining: 473ms\n",
            "982:\ttotal: 29.1s\tremaining: 443ms\n",
            "983:\ttotal: 29.1s\tremaining: 414ms\n",
            "984:\ttotal: 29.1s\tremaining: 384ms\n",
            "985:\ttotal: 29.1s\tremaining: 355ms\n",
            "986:\ttotal: 29.2s\tremaining: 325ms\n",
            "987:\ttotal: 29.2s\tremaining: 296ms\n",
            "988:\ttotal: 29.2s\tremaining: 266ms\n",
            "989:\ttotal: 29.3s\tremaining: 237ms\n",
            "990:\ttotal: 29.3s\tremaining: 207ms\n",
            "991:\ttotal: 29.3s\tremaining: 177ms\n",
            "992:\ttotal: 29.4s\tremaining: 148ms\n",
            "993:\ttotal: 29.4s\tremaining: 118ms\n",
            "994:\ttotal: 29.4s\tremaining: 88.7ms\n",
            "995:\ttotal: 29.4s\tremaining: 59.1ms\n",
            "996:\ttotal: 29.5s\tremaining: 29.6ms\n",
            "997:\ttotal: 29.5s\tremaining: 0us\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<catboost.core.CatBoostClassifier at 0x7f0f2368ce48>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 605
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sNEcNU-DiXpH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "clf2.save_model('call_center.model')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AeCbbIVhitsQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import files\n",
        "files.download('call_center.model') "
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}