categorical_dqn_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cfbf11bb",
   "metadata": {},
   "source": [
    "# Categorical DQN\n",
    "\n",
    "For ReLAx & Mujoco installation see: https://github.com/nslyubaykin/relax/blob/master/README.md\n",
    "\n",
    "Useful readings regarding Categorical DQN:\n",
    "\n",
    "- DQN paper: https://arxiv.org/pdf/1312.5602.pdf\n",
    "- Categorical DQN paper: https://arxiv.org/pdf/1707.06887.pdf\n",
    "\n",
    "!Note: For examining trained actor you may just load trained policies from \\trained_models directory. To do that instead of running 'Run Training and Validation loop' cell, just run 'Load Models' cell\n",
    "\n",
    "!Note: Trained policies, logs and videos are in the main repository of that notebook: https://github.com/nslyubaykin/relax_categorical_dqn_example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f78331dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/nikita/.local/lib/python3.6/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "import torch\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from tqdm import tqdm\n",
    "from inspect import getsource\n",
    "from torch.utils.tensorboard import SummaryWriter\n",
    "\n",
    "from relax.rl.critics import CategoricalDQN\n",
    "from relax.rl.actors import ArgmaxQValue\n",
    "\n",
    "from relax.zoo.critics import AtariDistributionalQCNN\n",
    "\n",
    "from relax.schedules import PiecewiseSchedule, LinearSchedule, CombinedSchedule, ConstantSchedule\n",
    "from relax.exploration import EpsilonGreedy\n",
    "\n",
    "from relax.gym.utils import get_wrapper_by_name, visualize_actor\n",
    "from relax.gym.wrappers.atari import *\n",
    "\n",
    "from relax.data.sampling import Sampler\n",
    "from relax.data.replay_buffer import ReplayBuffer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff9bf66c",
   "metadata": {},
   "source": [
    "# Constructing Distributional Discrete Deep Q-Network Critic\n",
    "\n",
    "Categorical DQN is a discrete action RL algorithm. That means that critic net is taking as an input an observations batch with a shape of (BATCH_SIZE, \\*obs.shape) (for Atari domain observation is a batch of stacked frames which then processed by CNN feature extractor) and outputting a a tensor of log probabilities with a shape (batch_size, acs_dim, n_atoms).\n",
    "\n",
    "!Note:\n",
    "            \n",
    "critic_net should have these attributes:\n",
    "\n",
    "- v_min - minimal support value of categorical distribution\n",
    "\n",
    "- v_max - maximal support value of categorical distribution\n",
    "\n",
    "- n_atoms - number of categorical distribution support elements\n",
    "\n",
    "- d_z - step of categorical distribution support =\n",
    "    (v_max-v_min)/(n_atoms-1)\n",
    "\n",
    "- support: torch.FloatTensor - categorical distribution support\n",
    "    which should be registered in model's buffer\n",
    "\n",
    "Therefore any custom user defined networks meeting this requirements should work well with `CategoricalDQN` critic\n",
    "\n",
    "So let's examine the default implementation of Categorical Atari Discrete Deep Q-Network Critic offered by ReLAx:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "894e1f13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "class AtariDistributionalQCNN(nn.Module):\n",
      "    \n",
      "    def __init__(self, in_channels, acs_dim,\n",
      "                 n_atoms, v_min, v_max):\n",
      "        \n",
      "        super(AtariDistributionalQCNN, self).__init__()\n",
      "        \n",
      "        # Categorical DQN specific params\n",
      "        self.n_atoms = n_atoms\n",
      "        self.v_min = v_min\n",
      "        self.v_max = v_max\n",
      "        self.d_z = (v_max-v_min)/(n_atoms-1)\n",
      "        self.register_buffer('support', torch.Tensor(n_atoms))\n",
      "        self.support.data.copy_(\n",
      "            torch.tensor(\n",
      "                [v_min + i*self.d_z for i in range(n_atoms)]\n",
      "            )\n",
      "        )\n",
      "\n",
      "        self.acs_dim = acs_dim\n",
      "        \n",
      "        self.net = nn.Sequential(\n",
      "            PreprocessImg(),\n",
      "            nn.Conv2d(in_channels=in_channels, out_channels=32, kernel_size=8, stride=4),\n",
      "            nn.ReLU(),\n",
      "            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2),\n",
      "            nn.ReLU(),\n",
      "            nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1),\n",
      "            nn.ReLU(),\n",
      "            nn.Flatten(),\n",
      "            nn.Linear(3136, 512),  # 3136 hard-coded based on img size + CNN layers\n",
      "            nn.ReLU(),\n",
      "            nn.Linear(512, acs_dim * n_atoms),\n",
      "        )\n",
      "        \n",
      "    def forward(self, x: torch.FloatTensor) -> torch.FloatTensor:        \n",
      "        out = self.net(x)\n",
      "        logits = out.view(-1, self.acs_dim, self.n_atoms)\n",
      "        log_probs = F.log_softmax(logits, dim=-1)\n",
      "        return log_probs\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(getsource(AtariDistributionalQCNN))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3baffda0",
   "metadata": {},
   "source": [
    "# Constructing ArgmaxQValue actor\n",
    "\n",
    "DQN uses a greedy non-parametric actor which simply selects an action with the biggest Q-value estimated by DQN critc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1107160",
   "metadata": {},
   "source": [
    "# Define Schedules for Training:\n",
    "\n",
    "Our Categorical DQN actor will not learn for the first 50k iterations. (Told to actor by passing 0 learning rate for the first 50k iterations)\n",
    "\n",
    "And it will also conduct maximum exploration during that phase to collect initial random training data.\n",
    "\n",
    "*!Note: we will construct our exploration schedule to be 30m timestamps long, however, for the time-saving purpose, we will run demonstration training only for 3m steps. In papers, DQN and its augmentations are trained for 200m steps, which may require several days of learning.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c52a375f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEICAYAAACavRnhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHatJREFUeJzt3X20HXV97/H3xxMDKppgSBGSQGIT1BOXip4iirUWUILXEquhJtUrtCjXFq5WvWpSe72IZK1GW1MpoELhirSaRLRylsamraD3ViXkcFE0gegxCEnEEvLkI+DB7/1jvgcmO/spZ/Z5yNmf11p7ZeY3v8c9O/u7Z+Z3ZhQRmJmZVfGE8e6AmZkd/hxMzMysMgcTMzOrzMHEzMwqczAxM7PKHEzMzKwyB5MuJ+lHks7sUF2fknRZp/MeYh/Ol/QfHazvDyVtl/RzSSd3qt4GbX1C0v8czTba7McJOd6ecWo/JM3vUF1fk/SWTue1gzmYTBKSXibpm5L2S9oj6RuSfme8+zWRtfnl8TfAxRFxVETc0aE2H8ov6wclfUHScQAR8baI+FDVNqqKiPtyvI9WqUfSYknflvTTHOvNkuZ1qp82sTiYTAKSngZ8Cfh74OnALOCDwMPj2a9J4kRg80gKNvllf3FEHAWcBEwHVo+wbxNWHll8Gng3MA2YB1wJVApQNnE5mEwOJwFExGcj4tGI+FVE/GtE3DmcQdJbJd0l6WeStkh6Yan8CyTdmUc1ayUdWSr3mvx1uS+PfJ5X2naypP+Xda4FyuUOOt3U7PRFs3bq5A1Jb5e0LX/xfkRS3c+ypJdK2pRj2yTppZm+Evhd4Io8SriiptwRkn4O9ADfkfTDTH9OHl3sk7RZ0jmlMp+S9HFJ6yX9Avj9RmMAiIg9wOeB55bKP3bqr8V7PyePanZJ2l3uv6Q/zX29V9IGSSdm+gcl/X0uP1HSLyR9JNeflEdMT5c0N9/jKbnt/HyvfybpHklvbNUW8ALgnoj4ahR+FhGfj4j7slyPpL+U9MOs93ZJc0pvz5mSfpBjv1KS2mgTSa+UdHfu7yuAcrlLJP1jaf2AcdZq1o7VERF+HeYv4GnAbuB64Gzg6Jrt5wI7gd+h+M81Hzgxt/0IuA04nuKo5i7gbbntZOAB4MUUX6rnZf4jgKnAvcA7gScCS4BfA5dl2fOB/6jpRwDzc/lTpbwN22kw3gBuyf6eAHwfeEttu7l9L/BfgSnAslyfkdu/NlyuyXtb7vMTgUHgL3P8pwM/A55VGtN+4DSKH2pH1qnvsTaBY4CbgRsO5T3J9e9QHNE8hSKIvyzLLc4+PifH/FfAN3Pb6cB3c/mlwA+BjaVt38nluTnuKVn/T0tjPA5Y2EZbzwQeyj7+PnBUzfvwHuC7wLMoPpPPL+2XoDjSnp77dxewqI02j8n9sST31TuBodL7fQnwj6U+PDbOOvumYTt+Nfi/Mt4d8KtDO7L40H8K2JH/gfqBY3PbBuAdDcr9CHhTaf3DwCdy+ePAh2rybwV+D3g58GNApW3fZGTBpGE7Dfocw18uuf7nwFdr26UIIrfVlP0WcH4uP/bl0eR9Lff5d4GfAE8obf8scElpTJ9uUd/XgF8C+ygC/D8BMw/lPQFeQvEFO6VO/V8BLiitPyHbOxF4EsUX/AxgOUVQ3AEcRXFa9PIsM5cDg8k+4PXAk9ptK9dPBdZlXx/K8R1VGsviJu/5y0rr64DlbYzvzcCtpW3K8Y0kmDQdm18Hv3yaa5KIiLsi4vyImE1x2uR44O9y8xyKX6GN/KS0/EuKLxco/oO+O0817JO0L+s6Pl87I/+npXtH2P1m7TSyvabdenmPr9OneymuKY3E8cD2iPhNk/q209rbI2J6RMyKiDdGxK46eZq9J3OAeyNiqEG5j5XK7KH4Up0VEb8CBnj8x8DXKX4AnJZpX6+tLCJ+AbwBeBtwv6QvS3p2q7ay7K0R8UcRMZMiEL8ceH+WrfKZbNTm8ZTe//xstrM/6mk6NjuYg8kkFBF3U/wKfG4mbQd+ewRVbQdW5hff8OvJEfFZ4H5gVvlcNsUpiWG/AJ48vCLpGSNsp5Hy+fUTKI6Sav2Y4kuBmrw7c/lQb5n9Y2BOzfWZcn0jqbORZu/JduCEBuf6twP/rabckyLim7n96xSntE4GNuX6WcApwP+p15GI2BARr6Q4xXU3cE2bbZXr2AR8gc58Jhu1eT+lz0V+NsufkwM+k0Crz2RbY7OCg8kkIOnZkt4taXauz6G4PnBrZvkH4H9IepEK89u8mHgN8DZJL85yT5H0XyQ9leJ00RDw9ryY+zqKL6Rh3wEWSnqBigv6l4ywnUbeI+noHOs7gLV18qwHTpL0x5KmSHoD0EtxPh7gPynO7bdrI8Wv5PfmmF8B/AGw5hDqaFez9+Q2ii/Ov870IyWdluU+AayQtBBA0jRJ55bq/TrF6aAtEfEIeWqH4mL5QUdIko5VMcX3KRSzA38ODB+ZNWxLxVT1t0r6rVx/NnAOB34mPyRpQY7veZJmtPG+NBvflyk+c6/LQPt2DgwY3wZeruLvaKYBK0bYjtXhYDI5/IziQu1GFbOIbgW+RzEtk4j4HLAS+Ezm/SLFxemmImIAeCtwBcWF60GKaxLkF9Hrcn0PxamQL5TKfh+4FPh34AdAwz8kbNZOEzcBt1N8QXwZuLZOvbuB11C8D7uB9wKviYgHM8vHgCU5W+fyFu0Nj/kPKCY5PAhcBbw5jwQ7qsV7/2j2Yz5wH8V1gTfktn8GVgFrJP2U4nNwdqnqb1JcOxk+CtlCcT2j7lEJxXfEuyiOyvZQnA77szba2kcRPL6rYlbcvwD/THFNDuCjFNdC/pXiAv+12a9W70vDNnO/ngv8NcX+XgB8o1T23yh+dNxJ8dn5Eg208T5aDR14ytts4pMUwIKIGBzvvphZwUcmZmZWmYOJmZlV5tNcZmZWmY9MzMyssrr3pJksjjnmmJg7d+54d8PM7LBy++23P5h/bNq2SR1M5s6dy8DAwHh3w8zssCLpkO9m4dNcZmZWmYOJmZlV5mBiZmaVOZiYmVllDiZmZlZZW8FE0iJJWyUNSlpeZ/sRKh73Oihpo6S5pW0rMn2rpLNa1Snp4kwLSceU0iXp8tx2pw587KyZmY2jlsFEUg9wJcUdM3uBZZJ6a7JdAOyNiPkUj+lclWV7gaXAQmARcJWKZz83q/MbwJkc/FCjsynuAroAuJDiSXRmZjYBtPN3JqcAgxGxDUDSGornI28p5VnM48+ruBG4Ih9MsxhYExEPA/dIGuTxZ17UrTMi7si02n4spngkagC3Spou6biIuP9QBjzaIoJPf+tedv/84fHuipl1sTOecyzPnzN9zNprJ5jM4sBHX+6geHZG3TwRMSRpP8Vzpmfx+MNwhssOP/ayVZ3t9GMWxUOCHiPpQoojF044ofzgv7Gxc9+v+F/9m7MvY968mRkAv/W0IydcMDmsRMTVwNUAfX19Y34Xy9/kM+j+9tzn8/oXzR7r5s3MxkU7F+B3cuBzlGdz4DOvD8iTj8ucRvGks0Zl26lzJP0wM7Nx0E4w2QQskDRP0lSKC+r9NXn6gfNyeQlwc17b6AeW5myveRQXz29rs85a/cCbc1bXqcD+iXa9xMysW7U8zZXXQC4GNgA9wHURsVnSpcBARPRTPL/5hrzAvociOJD51lFcrB8CLsrnV1Ovzkx/O8Wzup8B3ClpfUS8BVgPvJriWdi/BP6kU2+CmZlV09Y1k4hYT/FlXk77QGn5IeDcBmVXAivbqTPTLwcur5MewEXt9NfMzMaW/wLezMwqczDpsMCPQTaz7uNgYmZmlTmYjBL/waKZdRMHEzMzq8zBxMzMKnMwMTOzyhxMzMysMgeTDgvPDDazLuRgYmZmlTmYjBJPDTazbuJgYmZmlTmYmJlZZQ4mZmZWmYOJmZlV5mDSYZ4ZbGbdyMHEzMwqczAZJcJzg82seziYmJlZZQ4mZmZWmYOJmZlV5mBiZmaVOZh0WPi2wWbWhRxMzMysMgeTUeK7BptZN3EwMTOzyhxMzMysMgcTMzOrzMHEzMwqczDpME8MNrNu5GBiZmaVtRVMJC2StFXSoKTldbYfIWltbt8oaW5p24pM3yrprFZ1SpqXdQxmnVMz/QRJt0i6Q9Kdkl5dZeBmZtY5LYOJpB7gSuBsoBdYJqm3JtsFwN6ImA+sBlZl2V5gKbAQWARcJamnRZ2rgNVZ196sG+CvgHURcXLWedXIhmxmZp3WzpHJKcBgRGyLiEeANcDimjyLgetz+UbgDEnK9DUR8XBE3AMMZn1168wyp2cdZJ2vzeUAnpbL04AfH9pQzcxstLQTTGYB20vrOzKtbp6IGAL2AzOalG2UPgPYl3XUtnUJ8CZJO4D1wH+v11lJF0oakDSwa9euNoZnZmZVHU4X4JcBn4qI2cCrgRskHdT/iLg6Ivoiom/mzJlj3kkzs27UTjDZCcwprc/OtLp5JE2hOA21u0nZRum7gelZR21bFwDrACLiW8CRwDFt9H9M+abBZtaN2gkmm4AFOctqKsXF7/6aPP3Aebm8BLg5inux9wNLc7bXPGABcFujOrPMLVkHWedNuXwfcAaApOdQBBOfxzIzmwCmtMoQEUOSLgY2AD3AdRGxWdKlwEBE9APXUpx2GgT2UAQHMt86YAswBFwUEY8C1Kszm3wfsEbSZcAdWTfAu4FrJL2T4mL8+TGBHx4i3zbYzLpIy2ACEBHrKS56l9M+UFp+CDi3QdmVwMp26sz0bRSzvWrTtwCntdNfMzMbW4fTBXgzM5ugHEzMzKwyBxMzM6vMwaTjJuycADOzUeNgYmZmlTmYjBJPDDazbuJgYmZmlTmYmJlZZQ4mZmZWmYOJmZlV5mDSYRP3bmFmZqPHwcTMzCpzMBklvmmwmXUTBxMzM6vMwcTMzCpzMDEzs8ocTMzMrDIHkw7zzGAz60YOJmZmVpmDySiR7xtsZl3EwcTMzCpzMDEzs8ocTMzMrDIHEzMzq8zBpMN812Az60YOJmZmVpmDySjxXYPNrJs4mJiZWWUOJmZmVpmDiZmZVdZWMJG0SNJWSYOSltfZfoSktbl9o6S5pW0rMn2rpLNa1SlpXtYxmHVOLW37I0lbJG2W9JmRDtrMzDqrZTCR1ANcCZwN9ALLJPXWZLsA2BsR84HVwKos2wssBRYCi4CrJPW0qHMVsDrr2pt1I2kBsAI4LSIWAn8x4lGPovB9g82sC7VzZHIKMBgR2yLiEWANsLgmz2Lg+ly+EThDkjJ9TUQ8HBH3AINZX906s8zpWQdZ52tz+a3AlRGxFyAiHjj04ZqZ2WhoJ5jMAraX1ndkWt08ETEE7AdmNCnbKH0GsC/rqG3rJOAkSd+QdKukRfU6K+lCSQOSBnbt2tXG8EaHZwabWTc5nC7ATwEWAK8AlgHXSJpemykiro6Ivojomzlz5hh30cysO7UTTHYCc0rrszOtbh5JU4BpwO4mZRul7wamZx21be0A+iPi13nK7PsUwcXMzMZZO8FkE7AgZ1lNpbig3l+Tpx84L5eXADdHRGT60pztNY/iy/+2RnVmmVuyDrLOm3L5ixRHJUg6huK017ZDHK+ZmY2CKa0yRMSQpIuBDUAPcF1EbJZ0KTAQEf3AtcANkgaBPRTBgcy3DtgCDAEXRcSjAPXqzCbfB6yRdBlwR9ZN5n2VpC3Ao8B7ImJ39bfAzMyqahlMACJiPbC+Ju0DpeWHgHMblF0JrGynzkzfRjHbqzY9gHfla8LyXYPNrBsdThfgzcxsgnIwGSW+a7CZdRMHEzMzq8zBxMzMKnMwMTOzyhxMzMysMgeTDvPUYDPrRg4mZmZWmYPJqPHcYDPrHg4mZmZWmYOJmZlV5mBiZmaVOZiYmVllDiYdFnhusJl1HwcTMzOrzMFklPiuwWbWTRxMzMysMgcTMzOrzMHEzMwqczAxM7PKHEw6zHcNNrNu5GBiZmaVOZiMEs8MNrNu4mBiZmaVOZiYmVllDiZmZlaZg4mZmVXmYGJmZpU5mJiZWWUOJqNEvm2wmXURBxMzM6usrWAiaZGkrZIGJS2vs/0ISWtz+0ZJc0vbVmT6VklntapT0rysYzDrnFrT1uslhaS+kQzYzMw6r2UwkdQDXAmcDfQCyyT11mS7ANgbEfOB1cCqLNsLLAUWAouAqyT1tKhzFbA669qbdQ/35anAO4CNIxuumZmNhnaOTE4BBiNiW0Q8AqwBFtfkWQxcn8s3AmeouGiwGFgTEQ9HxD3AYNZXt84sc3rWQdb52lI7H6IINg8d4jjNzGwUtRNMZgHbS+s7Mq1unogYAvYDM5qUbZQ+A9iXdRzQlqQXAnMi4svNOivpQkkDkgZ27drVxvA6y3cNNrNudFhcgJf0BOCjwLtb5Y2IqyOiLyL6Zs6cOfqdMzOztoLJTmBOaX12ptXNI2kKMA3Y3aRso/TdwPSso5z+VOC5wNck/Qg4FeifyBfhPTHYzLpJO8FkE7AgZ1lNpbig3l+Tpx84L5eXADdHRGT60pztNQ9YANzWqM4sc0vWQdZ5U0Tsj4hjImJuRMwFbgXOiYiBEY7bzMw6aEqrDBExJOliYAPQA1wXEZslXQoMREQ/cC1wg6RBYA9FcCDzrQO2AEPARRHxKEC9OrPJ9wFrJF0G3JF1m5nZBNYymABExHpgfU3aB0rLDwHnNii7EljZTp2Zvo1itlez/ryinX6bmdnYOCwuwJuZ2cTmYNJhgecGm1n3cTAxM7PKHExGiW8abGbdxMHEzMwqczAxM7PKHEzMzKwyBxMzM6vMwaTDfNdgM+tGDiZmZlaZg8ko8dRgM+smDiZmZlaZg4mZmVXmYGJmZpU5mJiZWWUOJh3mmcFm1o0cTMzMrDIHk1EiPDfYzLqHg4mZmVXmYGJmZpU5mJiZWWUOJmZmVpmDSYeFbxtsZl3IwcTMzCpzMBktnhlsZl3EwcTMzCpzMDEzs8ocTMzMrDIHEzMzq8zBpMM8MdjMulFbwUTSIklbJQ1KWl5n+xGS1ub2jZLmlratyPStks5qVaekeVnHYNY5NdPfJWmLpDslfVXSiVUGbmZmndMymEjqAa4EzgZ6gWWSemuyXQDsjYj5wGpgVZbtBZYCC4FFwFWSelrUuQpYnXXtzboB7gD6IuJ5wI3Ah0c25LHhmcFm1k3aOTI5BRiMiG0R8QiwBlhck2cxcH0u3wicIUmZviYiHo6Ie4DBrK9unVnm9KyDrPO1ABFxS0T8MtNvBWYf+nDNzGw0tBNMZgHbS+s7Mq1unogYAvYDM5qUbZQ+A9iXdTRqC4qjla/U66ykCyUNSBrYtWtXy8GZmVl1h90FeElvAvqAj9TbHhFXR0RfRPTNnDlzbDtnZtalprSRZycwp7Q+O9Pq5dkhaQowDdjdomy99N3AdElT8ujkgLYknQm8H/i9iHi4jb6bmdkYaOfIZBOwIGdZTaW4oN5fk6cfOC+XlwA3R3H73H5gac72mgcsAG5rVGeWuSXrIOu8CUDSycAngXMi4oGRDdfMzEZDyyOTiBiSdDGwAegBrouIzZIuBQYioh+4FrhB0iCwhyI4kPnWAVuAIeCiiHgUoF6d2eT7gDWSLqOYwXVtpn8EOAr4XHGdnvsi4pzK70CH+Q70ZtaN2jnNRUSsB9bXpH2gtPwQcG6DsiuBle3UmenbKGZ71aaf2U5fJ4oMeGZmXeGwuwBvZmYTj4OJmZlV5mBiZmaVOZiYmVllDiZmZlaZg0nHeW6wmXUfB5NR4onBZtZNHEzMzKwyBxMzM6vMwcTMzCpzMDEzs8ocTMzMrDIHkw7zXYPNrBs5mIwS3zTYzLqJg4mZmVXmYGJmZpU5mJiZWWUOJmZmVpmDiZmZVeZg0mGeGWxm3cjBZJTI9w02sy7iYGJmZpU5mJiZWWUOJmZmVpmDiZmZVeZgYmZmlTmYdJjvGmxm3cjBZJT4rsFm1k0cTMzMrDIHEzMzq8zBxMzMKmsrmEhaJGmrpEFJy+tsP0LS2ty+UdLc0rYVmb5V0lmt6pQ0L+sYzDqntmrDzMzGV8tgIqkHuBI4G+gFlknqrcl2AbA3IuYDq4FVWbYXWAosBBYBV0nqaVHnKmB11rU3627YhpmZjb8pbeQ5BRiMiG0AktYAi4EtpTyLgUty+UbgCknK9DUR8TBwj6TBrI96dUq6Czgd+OPMc33W+/FGbUR0fjLuuk3bueb/bhtR2V8+8miHe2NmNvG1E0xmAdtL6zuAFzfKExFDkvYDMzL91pqys3K5Xp0zgH0RMVQnf6M2Hix3RNKFwIUAJ5xwQhvDO9j0Jz+RBcceNaKyAC/57Rk8d9a0EZc3MzvctBNMDisRcTVwNUBfX9+IjlpetfAZvGrhMzraLzOzyaydC/A7gTml9dmZVjePpCnANGB3k7KN0ncD07OO2rYatWFmZuOsnWCyCViQs6ymUlxQ76/J0w+cl8tLgJvzWkY/sDRnYs0DFgC3Naozy9ySdZB13tSiDTMzG2ctT3Pl9YmLgQ1AD3BdRGyWdCkwEBH9wLXADXmBfQ9FcCDzraO4WD8EXBQRjwLUqzObfB+wRtJlwB1ZN43aMDOz8afJ/OO+r68vBgYGxrsbZmaHFUm3R0TfoZTxX8CbmVllDiZmZlaZg4mZmVXmYGJmZpVN6gvwknYB946w+DHU/HV9F/CYu4PH3B2qjPnEiJh5KAUmdTCpQtLAoc5mONx5zN3BY+4OYz1mn+YyM7PKHEzMzKwyB5PGrh7vDowDj7k7eMzdYUzH7GsmZmZWmY9MzMysMgcTMzOrzMGkDkmLJG2VNChp+Xj3pxVJcyTdImmLpM2S3pHpT5f0b5J+kP8enemSdHmO705JLyzVdV7m/4Gk80rpL5L03SxzeT6WuWEbYzj2Hkl3SPpSrs+TtDH7uTYfcUA+BmFtpm+UNLdUx4pM3yrprFJ63c9BozbGaLzTJd0o6W5Jd0l6yWTfz5LemZ/r70n6rKQjJ9t+lnSdpAckfa+UNm77tVkbDUWEX6UXxS3xfwg8E5gKfAfoHe9+tejzccALc/mpwPeBXuDDwPJMXw6syuVXA18BBJwKbMz0pwPb8t+jc/no3HZb5lWWPTvT67YxhmN/F/AZ4Eu5vg5YmsufAP4sl/8c+EQuLwXW5nJv7uMjgHm573uafQ4atTFG470eeEsuTwWmT+b9TPG47nuAJ5Xe+/Mn234GXg68EPheKW3c9mujNpqOYaz+ExwuL+AlwIbS+gpgxXj36xDHcBPwSmArcFymHQdszeVPAstK+bfm9mXAJ0vpn8y044C7S+mP5WvUxhiNczbwVeB04Ev5wX8QmFK7LymenfOSXJ6S+VS7f4fzNfocNGtjDMY7jeKLVTXpk3Y/UwST7fkFOSX381mTcT8DczkwmIzbfm3URrP++zTXwYY/vMN2ZNphIQ/rTwY2AsdGxP256SfAsbncaIzN0nfUSadJG2Ph74D3Ar/J9RnAvogYyvVyPx8bW27fn/kP9b1o1sZomwfsAv63ilN7/yDpKUzi/RwRO4G/Ae4D7qfYb7czuffzsPHcr4f8PehgMolIOgr4PPAXEfHT8rYofl6M6jzwsWhjmKTXAA9ExO1j0d4EMYXiVMjHI+Jk4BcUpyYeMwn389HAYopAejzwFGDRWLQ9kRwO+9XB5GA7gTml9dmZNqFJeiJFIPmniPhCJv+npONy+3HAA5neaIzN0mfXSW/Wxmg7DThH0o+ANRSnuj4GTJc0/Djqcj8fG1tunwbs5tDfi91N2hhtO4AdEbEx12+kCC6TeT+fCdwTEbsi4tfAFyj2/WTez8PGc78e8vegg8nBNgELcibHVIqLeP3j3KemcmbGtcBdEfHR0qZ+YHhGx3kU11KG09+cMzZOBfbnoe4G4FWSjs5fhK+iOE98P/BTSadmW2+uqateG6MqIlZExOyImEuxj26OiDcCtwBL6vSn3M8lmT8yfWnOApoHLKC4WFn3c5BlGrUxqiLiJ8B2Sc/KpDOALUzi/UxxeutUSU/OPg2PedLu55Lx3K+N2mhsNC8oHa4vipkM36eY5fH+8e5PG/19GcXh6Z3At/P1aorzvl8FfgD8O/D0zC/gyhzfd4G+Ul1/Cgzm609K6X3A97LMFTx+94S6bYzx+F/B47O5nknxJTEIfA44ItOPzPXB3P7MUvn357i2krNcmn0OGrUxRmN9ATCQ+/qLFLN2JvV+Bj4I3J39uoFiRtak2s/AZymuCf2a4gj0gvHcr83aaPTy7VTMzKwyn+YyM7PKHEzMzKwyBxMzM6vMwcTMzCpzMDEzs8ocTMzMrDIHEzMzq+z/A1INMHEA9s1AAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr_schedule = PiecewiseSchedule({0: 50000}, 1e-4)\n",
    "lr_schedule.plot(int(1e6))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ab052675",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEVCAYAAAAb/KWvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3X+cXHV97/HXe3dmd3aTnfwgASEJBDVW0YcKRqTqVVr1XkCF3oo1tFZRlEtb1FavfdBeSxX1Xq3WWgtWQa2It/yQa21q40V6Ra0/QIIC5Ydg5IdJQEnI5tduktkfn/vHObM7O5nZnd3M/pgz7+fjsY/MzPnOOd8zZ/I+3/mec75HEYGZmWVLx3xXwMzMms/hbmaWQQ53M7MMcribmWWQw93MLIMc7mZmGeRwX4AkPSLplU2a1xclfajZZadZh/Mlfa+J8/uvkrZK2i/p5GbNdy5M9VlI+oakN8/CctdKCkm5I5zPtyW9rUl1er+kLze7rCUc7rNI0ksl/UDSHkm7JH1f0gvnu14LWYPh8XHg4ohYHBE/adJy/4uk70raJ2mHpO9IOrsZ856OiDgzIq6e7eVIerakb6bfy92S7pB01mwv1+aOw32WSCoCXwf+DlgOrAI+AByaz3plxAnAvTN5o6TOGq+dC3wF+BKwGjgGuBR47RHUcaH7F+Bm4CnA0cA7gb3zWiNrKof77HkGQERcGxEjEXEgIr4ZEXeXC0h6u6T709bifZJOqXj/8yXdnbb6r5dUqHjfayTdmba4fiDpuRXTTpb043Se1wOV7zusSyD9qf70Wisw2XJqlA1J75T0kKSdkj4mqeb3S9KLJd2ertvtkl6cvv5h4D8Bl6ddLpdXva9b0n6gE7hL0s/T15+Vtvh3S7q3ssWddjX9vaRNkgaA36iap4BPAB+MiM9FxJ6IGI2I70TE29MyHZLeJ+lRSU9I+pKkJem0cnfHW9Kuon5JF0l6Ybr9dlevR7rYy9P1/6mkV1RMGPvlUt5ekj6ezvdhSWdWlF0i6fOSHpe0XdKHyjsvSZ3p+3ZKegh4dcX7VgAnAldFRCn9+35EfK+izDnptt8r6eeSzqio/wnpr9B9aet/RcX7Tku/K7sl3SXp9IppJ6a/iPZJuhmofN/pkrZVfUh1uycnW46lIsJ/s/AHFIEngauBM4FlVdNfD2wHXggIeDpwQjrtEeBHwHEkrf77gYvSaScDTwAvIgm5N6flu4Eu4FHgT4A8cC4wBHwofe/5wPeq6hHA09PHX6woW3c5ddY3gFvS+h4PPAi8rXq56fR+4PeBHHBe+vyodPq3y++b5LOtrHMe2AL8ebr+vwnsA36tYp32AC8hacwUqub1zHR+J06yvLemy3gqsBj4KnBNOm1t+v7PkOxI/zNwEPgaSYt4Vfo5vrzisxiu2EZvSOu3vHr907JDwNvTbfAHwGOA0un/BHwWWJQu60fAf0unXQT8FFiTfua3pPXMkXzffkbyy/K3gGOq1vfUtE6vSj+zVcAzK+r3c5LGS0/6/CPptFUk3/mz0ve9Kn2+Mp3+Q5IdaTfwsnQ7fTmddjqwraoejwCvTB+/v6LspMvxX/LnlvssiYi9wEtJ/kNdBeyQtFHSMWmRtwF/FRG3R2JLRDxaMYtPRcRjEbGL5Cf089PXLwQ+GxG3RfKL4GqSrp7T0r888MmIGIqIG4HbZ7gKky2nno9GxK6I+AXwSZLgrvZq4GcRcU1EDEfEtSQhNNMukNNIAvcjkbRAv0USWpXL/udIWqajEXGw6v1Hpf8+Pskyfg/4REQ8FBH7gT8DNmjiwckPRsTBiPgmMABcGxFPRMR24N9JdpZlTzC+ja4HHqCiZV3l0Yi4KiJGSBoKxwLHpN+js4A/joiBiHgC+BtgQ/q+30mXsTX9Dv2v8gwjScjfIAnPvwYeV3K8YV1a5ALgCxFxc/qZbY+In1bU6R8i4sGIOADcwPh3843ApojYlL7vZmAzcJak40kaMn8REYci4rsk3+uZqLucGc4vkxzusygi7o+I8yNiNfAckpb4J9PJa0haQPX8suLxIEmAQdLf/J705+huSbvTeR2X/m1P//OWVe4wpmOy5dSztWq5tcoeV6NOj5K0xmbiOGBrRIxOMr+t1Pdk+u+xUyyjss6PkrSAj6l47VcVjw/UeL644nmtbVTvcx37HkTEYPpwMcn2yZMEc3n7fJakBV+uc/X2GBMR2yLi4oh4WjqvAZJjDnBk383XV31nXkry2R4H9EfEQL06TcNky7GUw32OpC2fL5KEPCT/8Z42g1ltBT4cEUsr/nrTFvDjwKq0H7ns+IrHA0Bv+Ymkp8xwOfWsqVruYzXKPEbyn5OqstvTx9MdpvQxYE1V/37l/Kaa5wMk6/q6KZZRWefjSbpWflW7+JRqbaNan9VktpL8klpRsX2KEfHsdPrjHL49aoqIrcAVNOe7eU3Vd2ZRRHwkrc8ySYvq1Kn6u9kJrJzBcizlcJ8lkp4p6T2SVqfP15B0FdyaFvkc8N8lvUCJp0uqDr1argIukvSi9H2LJL1aUh9Jn+Yw8E5JeUm/TdJ/WnYX8GxJz1dygPb9M1xOPe+VtCxd13cB19coswl4hqTflZST9AbgJJKuFEgC86lTfwxjbiNpPf5pus6nk3TxXNfIm9MW9LuBv1ByULSo5ADqSyVdmRa7FviT9IDgYuB/AtdHxPA06lnpaMa30euBZ5F8Lg2LiMeBbwJ/XVHnp0l6eVrkhnQZqyUtAy4pvzfdRh9Iv3Md6QHRtzL+3fw88BZJr0inr5L0zAaq9WXgtUpOK+2UVEgPlK5Ouxw3Ax+Q1CXppUzsinsQKKTfsTzwPpK++Wktp5HPrl043GfPPpKDkbcpOUvjVuAe4D0AEfEV4MPAP6Zlv0Zy4GtSEbGZ5ADb5SQHIreQHHgjIkrAb6fPd5EcrPtqxXsfBC4D/o3kgFrdi2kmW84k/hm4A7gT+FeSkKie75PAa0g+hyeBPwVeExE70yJ/C5yr5OyQT02xvPI6v5bkoPVO4NPAm6r6iKeax40kn9VbSVrQvwI+lK4PwBeAa4DvAg+THDB9R6Pzr+E2YF1a3w8D56afy3S9ieQg8n0k2+hGxrsmrgJuItmh/5iK7wFQIjkQ/G8kpz/eQ/Ir4HyAiPgR8BaSPvw9wHc4/NfWYdJfAOeQHNzeQdLCfi/jOfO7JP8ndgF/yXg3EBGxB/hDkkbPdpKW/ISzZ6axHGP8qLvZEZEUwLqI2DLfdTEz7+nMzDLJ4W5mlkHuljEzyyC33M3MMuiIhv88EitWrIi1a9fO1+LNzFrSHXfcsTMi6l0DMGbewn3t2rVs3rx5vhZvZtaSJDV0Za+7ZczMMsjhbmaWQQ53M7MMcribmWWQw93MLIOmDHdJX1Bya7F76kyXpE9J2qLktmKn1CpnZmZzp5GW+xeBMyaZfibJCHfrSO7e8/dHXi0zMzsSU4Z7ejusXZMUOQf4UnqruFuBpZJm7Y4oP/5FP5d/62fs3H9othZhZtbymtHnvoqJt/PaRp1bpkm6UNJmSZt37Ngxo4X96OFdfPybD3L1Dx6Z0fvNzNrBnB5QjYgrI2J9RKxfuXLKq2druujlT2NJT57+wVKTa2dmlh3NCPftTLxX42om3r+y6Zb25tl3cKZ3ODMzy75mhPtG4E3pWTOnAXvS+zvOmmIhz94DQ7O5CDOzljblwGGSrgVOB1ZI2kZy78M8QER8huTGvmeR3GNzkOTei7Oqr5Bzy93MbBJThntEnDfF9AD+qGk1akBfIccjOwfncpFmZi2lJa9QLRby7D3obhkzs3paMtz7Cj6gamY2mRYN9xz7Dw0zMur7v5qZ1dKS4V7syQOw3613M7OaWjLc+wrJcWD3u5uZ1daS4V5Mw9397mZmtbVouCfdMm65m5nV1pLh3peGu1vuZma1tWi4l7tl3HI3M6ulJcO9fLaMx5cxM6utJcO9zwdUzcwm1ZLhnu/soJDvYN8hh7uZWS0tGe6QHFR1t4yZWW0tG+5FD/trZlZXy4Z7n0eGNDOrq4XD3S13M7N6Wjbciz1uuZuZ1dO64e6Wu5lZXS0b7skNO9xyNzOrpWXDvVjIcXBolNLw6HxXxcxswWnZcB8fPMytdzOzai0c7h6CwMysnpYNd4/pbmZWX8uGu1vuZmb1tXC4u8/dzKyelg33Yk96k+wDbrmbmVVr2XDvc5+7mVldLRvui7vd525mVk/Lhntnh+jrzrnlbmZWQ8uGO3hkSDOzelo83D2+jJlZLQ2Fu6QzJD0gaYukS2pMP17SLZJ+IuluSWc1v6qHK/bkfLaMmVkNU4a7pE7gCuBM4CTgPEknVRV7H3BDRJwMbAA+3eyK1tJXyLPvkFvuZmbVGmm5nwpsiYiHIqIEXAecU1UmgGL6eAnwWPOqWJ/73M3Mamsk3FcBWyueb0tfq/R+4I2StgGbgHfUmpGkCyVtlrR5x44dM6juRH2FHHsPuOVuZlatWQdUzwO+GBGrgbOAayQdNu+IuDIi1kfE+pUrVx7xQouFPPsODhMRRzwvM7MsaSTctwNrKp6vTl+rdAFwA0BE/BAoACuaUcHJ9BXyDI8GB4d8ww4zs0qNhPvtwDpJJ0rqIjlgurGqzC+AVwBIehZJuB95v8sUxkeGdNeMmVmlKcM9IoaBi4GbgPtJzoq5V9Jlks5Oi70HeLuku4BrgfNjDvpKij0eX8bMrJZcI4UiYhPJgdLK1y6teHwf8JLmVm1q5Zb7Xp8xY2Y2QUtfoVr0DTvMzGpq8XBPu2V8OqSZ2QQtHe7jd2Nyy93MrFKLh7vPljEzq6Wlw723q5PODvlsGTOzKi0d7pI8voyZWQ0tHe7gwcPMzGpp+XAvFvI+W8bMrErLh7tb7mZmh8tAuOd9QNXMrErLh3t52F8zMxvX8uHeV8i55W5mVqXlw71YyLH/0DCjo75hh5lZWeuHe0+eCNhfcteMmVlZy4d7n0eGNDM7TAbCvTx4mPvdzczKWj7cx4f9dcvdzKys5cPdI0OamR0uQ+HulruZWVkGwt03yTYzq5aBcHfL3cysWsuHeyHfSVeuwy13M7MKLR/ukFyl6pa7mdm4jIS7x3Q3M6uUiXD3mO5mZhNlJNzzPs/dzKxCJsK92JNjr1vuZmZjMhHufd1uuZuZVcpGuLvP3cxsgkyEe7Enz2BphKGR0fmuipnZgtBQuEs6Q9IDkrZIuqROmd+RdJ+keyX9Y3OrObnyVar73Xo3MwMgN1UBSZ3AFcCrgG3A7ZI2RsR9FWXWAX8GvCQi+iUdPVsVrmV8TPdhli3qmstFm5ktSI203E8FtkTEQxFRAq4Dzqkq83bgiojoB4iIJ5pbzckV05a7hyAwM0s0Eu6rgK0Vz7elr1V6BvAMSd+XdKukM2rNSNKFkjZL2rxjx46Z1bgGjwxpZjZRsw6o5oB1wOnAecBVkpZWF4qIKyNifUSsX7lyZZMW7ZEhzcyqNRLu24E1Fc9Xp69V2gZsjIihiHgYeJAk7OfEkp7yrfbccjczg8bC/XZgnaQTJXUBG4CNVWW+RtJqR9IKkm6ah5pYz0m55W5mNtGU4R4Rw8DFwE3A/cANEXGvpMsknZ0Wuwl4UtJ9wC3AeyPiydmqdLXF3Q53M7NKU54KCRARm4BNVa9dWvE4gHenf3Mu19nBoq5OH1A1M0tl4gpV8MiQZmaVMhTuHl/GzKwsU+Hubhkzs0Rmwr3Yk3fL3cwslZlwT/rcHe5mZpCpcM/5IiYzs1Rmwr3olruZ2ZjMhHtfIUdpZJSDQyPzXRUzs3mXmXAveggCM7Mx2Qn3Hg/7a2ZWlplw9+BhZmbjMhTu5VvtueVuZpaZcC+W78Z0wC13M7PMhPt4t4xb7mZmGQx3t9zNzDIT7ou6cnTIZ8uYmUGGwr2jQyzu9rC/ZmaQoXCH5IwZt9zNzDIW7sWevM+WMTMjY+Ge3I3JLXczs0yFe9G32jMzAzIX7u5zNzODjIW7b5JtZpbIWLjn2XdwiIiY76qYmc2rTIV7sSfHaMBAyTfsMLP2lqlw98iQZmaJjIW7x5cxM4PMhXt52F+33M2svWUq3H0fVTOzRKbCfazl7j53M2tzmQp3t9zNzBINhbukMyQ9IGmLpEsmKfc6SSFpffOq2Lhij1vuZmbQQLhL6gSuAM4ETgLOk3RSjXJ9wLuA25pdyUZ15zrId8otdzNre4203E8FtkTEQxFRAq4DzqlR7oPAR4GDTazftEgau0rVzKydNRLuq4CtFc+3pa+NkXQKsCYi/nWyGUm6UNJmSZt37Ngx7co2oljIeUx3M2t7R3xAVVIH8AngPVOVjYgrI2J9RKxfuXLlkS66JrfczcwaC/ftwJqK56vT18r6gOcA35b0CHAasHG+Dqp6ZEgzs8bC/XZgnaQTJXUBG4CN5YkRsSciVkTE2ohYC9wKnB0Rm2elxlPwmO5mZg2Ee0QMAxcDNwH3AzdExL2SLpN09mxXcLrccjczg1wjhSJiE7Cp6rVL65Q9/cirNXNJn7vD3czaW6auUIVkTPf9h4YZGfUNO8ysfWUu3Mvjy+x3693M2lgGwz3pafJBVTNrZ5kL96JHhjQzy2K4e2RIM7PMhfv4fVQd7mbWvjIX7sWetM/dt9ozszaWuXAfb7k73M2sfWUw3N3nbmaWuXDPd3ZQyHf4bBkza2uZC3dITod0y93M2lkmw92Dh5lZu8touHvYXzNrb5kM92JPnr1uuZtZG8tkuCfdMm65m1n7ymS4F93nbmZtLqPhnvcVqmbW1jIZ7n2FHIeGRykNj853VczM5kVGw91DEJhZe8tkuI8NHuZ+dzNrU5kM975ut9zNrL1lM9w9eJiZtblMhnuxJ73Vns+YMbM2lclwd8vdzNpdRsPdN8k2s/aWzXDvziH5bBkza1+ZDPeODrG4y+PLmFn7ymS4g8d0N7P2ltlwL/Z4fBkza1+ZDXe33M2snTUU7pLOkPSApC2SLqkx/d2S7pN0t6T/J+mE5ld1evoKefYdcsvdzNrTlOEuqRO4AjgTOAk4T9JJVcV+AqyPiOcCNwJ/1eyKTlexkGPvAbfczaw9NdJyPxXYEhEPRUQJuA44p7JARNwSEYPp01uB1c2t5vT1FfI+W8bM2lYj4b4K2FrxfFv6Wj0XAN+oNUHShZI2S9q8Y8eOxms5A+U+94iY1eWYmS1ETT2gKumNwHrgY7WmR8SVEbE+ItavXLmymYs+TF8hz/BocGBoZFaXY2a2EDUS7tuBNRXPV6evTSDplcD/AM6OiEPNqd7Mlcd09xkzZtaOGgn324F1kk6U1AVsADZWFpB0MvBZkmB/ovnVnD7fjcnM2tmU4R4Rw8DFwE3A/cANEXGvpMsknZ0W+xiwGPiKpDslbawzuzlTHhnS48uYWTvKNVIoIjYBm6peu7Ti8SubXK8jVix4THcza1+ZvUK16DHdzayNZTbcx/vcHe5m1n4yG+7ls2V8ww4za0eZDfeefCedHfLZMmbWljIb7pI8MqSZta3MhjskZ8z4bBkza0eZDne33M2sXTnczcwyKNPhXizkfbaMmbWlTId7Mqa7W+5m1n4yHu45t9zNrC1lOtyLPXn2HxpmdNQ37DCz9pLtcC/kiID9JXfNmFl7yXS493nwMDNrUw0N+duqysP+vu+f/oM1y3tZ1tvF8kVdLO3Ns3xR19jz5Yu6KOQ757m2ZmbNk+lwf/7xS/n1px7Fo08O8pOtu9k9WP/gak++k2W9eZalYV8O/mW9XSxblJ/wvLyD8A7BzBaqTIf7sUt6uPbC08aeD4+MsufAEP2DJXYNJP/2D5TYVf63/Npgia27Btk1UJr0Tk6LujpZWg79RV0sL+8certYmv67bFHyK2F5bxdLe7voymW6J8zMFohMh3u1XGcHRy3u5qjF3Q2/Z2hklN2D4zuCyh3DroGKncPgEI/sHKB/oMS+Q/V3CIu7c0ng93aN7QiWLeoa/9VQfj39lbC0N0++0zsEM5uetgr3mch3drCyr5uVfY3vEErDo+weLP8iqLEjGEh2BrsGSmx5Yj/9AyUGSiN159dXyB3eVVSjC2n5ojxLe7tY2pMn5x2CWVtzuM+CrlwHRxcLHF0sNPyeg0MjE34h1Ooq2jVQ4ol9B3ngl/vYNVDiwFD9HcKSnvJB43zFsYPyjqDiGEL62pKePJ0dasbqm9kC4HBfIAr5Tp6ypJOnLJneDmH8V8FQxQ6hlP5yGKJ/oMRjuw9y72N72TVQ4tDwaM15SbC0Jz95V1HF68sXdVEs5OnwDsFsQXK4t7BCvpNjl/Rw7JKehspHBAeGRuhPQ3/XQGlCl1H/4PgOYuuuQe7eluw0SiO1dwgdYuwXwfLeilNMJxw7SLqKys+LhRySdwhms83h3kYk0duVo7crx6qlje8QBksjE3cE6UHl3YMTdxDjp5yWGBqpPeRDrkPpGUb59NjB+E6guquofLbR4m7vEMymy+Fuk5LEou4ci7pzrFne29B7IoL9h4YP6yoaP3YwNHZc4ec79tP/aHJcYaTOGED5TtW9AK3e671dnd4hWFtzuFvTJfevzdNXyHP8UY3tEEZHg32HhqsOJpfYPXj4DuKBX+4bO/hcb0y4rlxH/a6i9FhC5S+F5b1d9HT5ojTLDoe7LQgdHWJJT54lPXnWsqih94yOBnsPDk3oKqp3ttH9j++lf6DE7gNDRJ0dQiHfMXax2WEXpi3qqjh2MP4rwVcp20LlcLeW1ZH23y/t7Wr4PSOjwZ4D4zuEehem9Q+W2L77ALsGSuyZ5CbrvV2ddYeoqHVh2tLePN057xBs9jncra10dmhssLhGDY+MsvtA5RlG9S9M+0U6bMVkI5Eu7s7VOXaQH//VMPbrIdlp+Cplmy6Hu9kUcp0drFjczYppDFtRGh5l94HxK5TrXZjWP1DioZ376R8YYv8kw1b0deeS4wRVYxjVuzDNVymbw91sFnTlOji6r8DRfY1flHZoePwq5VoXpvWnYxjt3F/iwV/tp3+wxOAkw1YUy8NWNDCG0fJFvko5axzuZgtEd66TY4qdHDPNYSv6q8cwqrowrX+wxC/3HuT+x/fy5BRXKS/pydf8RVB9YVr5GoUlPb5KeaFqKNwlnQH8LdAJfC4iPlI1vRv4EvAC4EngDRHxSHOrambVpnuVMsCB0kjNHcGuwYkXpm3ffYB7tu9h12CJUp0dQodgaW/VGEZTXJjWV8h5hzAHpgx3SZ3AFcCrgG3A7ZI2RsR9FcUuAPoj4umSNgAfBd4wGxU2syPT09VJT1cPx03jKuUDQyOTjGGUvj5Q4he7Brlz6276J7lKubNDSfdQ1VhFdS9MW9RFn69SnrZGWu6nAlsi4iEASdcB5wCV4X4O8P708Y3A5ZIUUe+MYjNrFZXDVqxe1th7IoKB0sjYTmCyC9Me3jnAHY8mw1YM17kqLdehsbGJOjIQ8u98xTpe+7zjZnUZjYT7KmBrxfNtwIvqlYmIYUl7gKOAnZWFJF0IXAhw/PHHz7DKZrbQSWJxd47F0xy2Yuwq5TpjGO09MEzQ+m3GJT35WV/GnB5QjYgrgSsB1q9f3/pbyMyaRhLFQp5iIc8JRzV2lbLV18iJsNuBNRXPV6ev1SwjKQcsITmwamZm86CRcL8dWCfpREldwAZgY1WZjcCb08fnAt9yf7uZ2fyZslsm7UO/GLiJ5FTIL0TEvZIuAzZHxEbg88A1krYAu0h2AGZmNk8a6nOPiE3ApqrXLq14fBB4fXOrZmZmM+XBJ8zMMsjhbmaWQQ53M7MMcribmWWQ5uuMRUk7gEdn+PYVVF392sK8LgtPVtYDvC4L1ZGsywkRsXKqQvMW7kdC0uaIWD/f9WgGr8vCk5X1AK/LQjUX6+JuGTOzDHK4m5llUKuG+5XzXYEm8rosPFlZD/C6LFSzvi4t2eduZmaTa9WWu5mZTcLhbmaWQQs63CWdIekBSVskXVJjerek69Ppt0laO/e1bEwD63K+pB2S7kz/3jYf9ZyKpC9IekLSPXWmS9Kn0vW8W9Ipc13HRjWwLqdL2lOxTS6tVW6+SVoj6RZJ90m6V9K7apRpie3S4Lq0ynYpSPqRpLvSdflAjTKzl2ERsSD/SIYX/jnwVKALuAs4qarMHwKfSR9vAK6f73ofwbqcD1w+33VtYF1eBpwC3FNn+lnANwABpwG3zXedj2BdTge+Pt/1bGA9jgVOSR/3AQ/W+H61xHZpcF1aZbsIWJw+zgO3AadVlZm1DFvILfexG3NHRAko35i70jnA1enjG4FXaGHeIr2RdWkJEfFdkjH76zkH+FIkbgWWSjp2bmo3PQ2sS0uIiMcj4sfp433A/ST3Na7UEtulwXVpCelnvT99mk//qs9gmbUMW8jhXuvG3NUbecKNuYHyjbkXmkbWBeB16U/mGyWtqTG9FTS6rq3i19Of1d+Q9Oz5rsxU0p/1J5O0Eiu13HaZZF2gRbaLpE5JdwJPADdHRN3t0uwMW8jh3m7+BVgbEc8FbmZ8b27z58ck43g8D/g74GvzXJ9JSVoM/B/gjyNi73zX50hMsS4ts10iYiQink9y7+lTJT1nrpa9kMM9SzfmnnJdIuLJiDiUPv0c8II5qluzNbLdWkJE7C3/rI7kbmR5SSvmuVo1ScqThOH/joiv1ijSMttlqnVppe1SFhG7gVuAM6omzVqGLeRwz9KNuadcl6r+z7NJ+hpb0UbgTenZGacBeyLi8fmu1ExIekq5/1PSqST/XxZc4yGt4+eB+yPiE3WKtcR2aWRdWmi7rJS0NH3cA7wK+GlVsVnLsIbuoTofIkM35m5wXd4p6WxgmGRdzp+3Ck9C0rUkZyuskLQN+EuSA0VExGdI7rV7FrAFGATeMj81nVoD63Iu8AeShoEDwIYF2nh4CfD7wH+k/bsAfw4cDy23XRpZl1bZLscCV0vqJNkB3RARX5+rDPPwA2ZmGbSQu2XMzGyGHO5mZhnkcDczyyCHu5lZBjnczczmwFQD1VWV/ZuKgdEelLR72svz2TJmZrNP0suA/SRj/DR8paqkdwAnR8Rbp7M8t9zNzOZArYHqJD1N0v+VdIekf5f0zBpvPQ+4drrLW7AXMZmZtYErgYsi4meSXgR8GvjN8kRJJwAnAt+a7ozzeoZlAAAAn0lEQVQd7mZm8yAdHO3FwFcqRvntriq2AbgxIkamO3+Hu5nZ/OgAdqejRtazAfijmc7czMzmWDqU8cOSXg9jt0J8Xnl62v++DPjhTObvcDczmwPpQHU/BH5N0jZJFwC/B1wg6S7gXibeoW0DcN1MB0XzqZBmZhnklruZWQY53M3MMsjhbmaWQQ53M7MMcribmWWQw93MLIMc7mZmGfT/AUpwawGDlwWBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "schedule1 = ConstantSchedule(1)\n",
    "schedule2 = LinearSchedule(1.04, 0, 1200000)\n",
    "schedule1 = CombinedSchedule(schedule1, schedule2, min)\n",
    "schedule2 = LinearSchedule(0.1, 0, 25000000)\n",
    "eps_schedule = CombinedSchedule(schedule1, schedule2, max)\n",
    "eps_schedule.plot(int(3e7))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1301e0fa",
   "metadata": {},
   "source": [
    "# Create training and eval environments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e50bf275",
   "metadata": {},
   "source": [
    "On practice, when training DQN on Atari 2600 domain, algorithm is not fed with raw environment observations, instead several wrappers are applied:\n",
    "\n",
    "1) EpisodicLifeEnv - make end-of-life == end-of-episode, but only reset on true game over. Done by DeepMind for the DQN and co. since it helps value estimation\n",
    "\n",
    "2) NoopResetEnv - Sample initial states by taking random number of no-ops on reset. No-op is assumed to be action 0.\n",
    "\n",
    "3) MaxAndSkipEnv - Return only every skip-th frame and do max-pooling over most recent two observations\n",
    "\n",
    "4) FireResetEnv - Take action on reset for environments that are fixed until firing.\n",
    "\n",
    "5) ProcessFrame84 - Crop valid region of an observation, convert to grayscale, resize to 84x84. Cast to uint8 dtype for lower memory usage.\n",
    "\n",
    "6) ClipRewardEnv - Bin reward to {+1, 0, -1} by its sign."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ff483d93",
   "metadata": {},
   "outputs": [],
   "source": [
    "def makeenv(env_name, hcrop, hstretch, seed):\n",
    "    \n",
    "    env = gym.make(env_name)\n",
    "    ob_unwrap = env.reset()\n",
    "    \n",
    "    env = gym.wrappers.Monitor(\n",
    "        env,\n",
    "        'monitor_train_logs',\n",
    "        force=True,\n",
    "        video_callable=None,\n",
    "    )\n",
    "    \n",
    "    env = wrap_deepmind(\n",
    "        env, \n",
    "        skip=4,\n",
    "        hcrop=hcrop, \n",
    "        hstretch=hstretch\n",
    "    )\n",
    "    \n",
    "    env.seed(seed)\n",
    "    \n",
    "    ob = env.reset()\n",
    "    \n",
    "    return env, ob_unwrap, ob "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "80977322",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(210, 160, 3) (84, 84, 1)\n"
     ]
    }
   ],
   "source": [
    "env_name = 'Breakout-v0'\n",
    "\n",
    "# Capture game region by adjusting these params:\n",
    "hcrop=13 # skip n top lines of pixels\n",
    "hstretch=100 # compression of vertical dimention from 210 -> hstretch (should be > than hcrop+84)\n",
    "\n",
    "env, ob_unwrap, ob = makeenv(\n",
    "    env_name=env_name, hcrop=hcrop, \n",
    "    hstretch=hstretch, seed=12\n",
    ")\n",
    "\n",
    "eval_env, _, _ = makeenv(\n",
    "    env_name=env_name, hcrop=hcrop, \n",
    "    hstretch=hstretch, seed=27\n",
    ")\n",
    "\n",
    "\n",
    "print(ob_unwrap.shape, ob.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "507de0bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD3CAYAAAAT+Z8iAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHdRJREFUeJzt3X20XVV97vHvY3hpjSiBIIMmwQRN6UBHDZoirYgUiqJSQYcXobaiYiO30NKLtwrYq9xWW9qrKB1avVEoeK+8iaA0gyKUYJCrRBLMwJCICTGQhJcQwlsDFRKe+8eaB3YO5+S87L3P2nvl+Yyxx15rrjXX+p2dk9+Ze6615pRtIiKiuV5SdwAREdFdSfQREQ2XRB8R0XBJ9BERDZdEHxHRcEn0ERENl0QfEdFwSfRdIGmtpKcl/YekByVdLOlldccVETunJPru+UPbLwPmAAcDZ9ccT0TspJLou8z2g8D3qRI+AJLeJemnkp6QtE7SuS3bLpH08bI8TZIlnVbWXy1ps6Qh/90kfUTSSkmPSvq+pFe1bLOkUyWtkvSYpK+osntZf13LvvuUbySv7PgHEhETLom+yyRNB94BrG4p3gJ8ENgTeBfwXyUdX7YtAo4oy28F1gCHt6z/0PZzQ5znOOAc4L3APsAPgcsG7XYs8DvAbwMnAG+3/SvgauCklv1OABbZ3jjGHzcielASffd8V9KTwDpgI/CZgQ22f2D7Z7afs30nVUJ+a9m8CDistNoPB/4ReHPZ9tayfSinAn9ve6XtrcDfAXNaW/XAebYfs30fcDMvfMu4FDixZb8/KmUR0QBJ9N1zvO09qFrnvwVMHdgg6U2Sbpb0sKTHqZL0VADb91C1+OcAbwEWAPdLOpAdJ/pXAReUbpjHgM2AgGkt+zzYsvwUMHCB+GbgpSWumeXc14zz546IHpNE32W2FwEXA59vKb4UuBaYYfsVwNeokvKARcD7gN1sbyjrJwNTgGXDnGod8DHbe7a8ft32j0YR4zbgSqrum5OABbafHMOPGRE9LIl+YnwJOFrS68v6HsBm2/8p6RCqrpJWi4DTgVvK+g/K+q0lKQ/la8DZkl4LIOkVkv7LGGK8FHg/8AHSbRPRKEn0E8D2w8A3gU+Xoj8D/qb04X+aqjXdahHVH4OBRH8r8NKW9aHOcQ3wD8Dlkp4AllNdBB5tjIupuox+A/i30daLiN6nTDwSEdFsadFHRDRcEn1ERMN1LdFLOkbS3ZJWSzqrW+eJiIgd60ofvaRJwC+Ao4H1wO3ASbZXdPxkERGxQ91q0R8CrLa9xvYzwOXAcV06V0RE7MAuXTruNKoHeAasB9403M6ScutPdNsm2/vUHUREHbqV6EckaR4wr67zx07n3roDiKhLtxL9BmBGy/r0UvY82/OB+ZAWfUREN3Wrj/52YLakWZJ2oxoZ8dounSsiInagKy1621slnU414cYk4CLbd3XjXBERsWM9MQRCr3TdnH/++WPa/8wzzxx3/U7WbVed595RHB0+11Lbczt5wIh+kSdjIyIarra7bvpBO63uwfUn8ttCO7rZYo+IeqRFHxF9S9LFkj5bdxy9Li362M5I3xzS4u9vks4GDrf9jpayVVRPsg8u+x+2L68hzOiwJPrYYfKeqC6jmDC3AGdJmmR7m6T9gF2BgweVvYYhJrqRtEuZfD76SLpuInYut1Ml9jll/S1Uk8PfPajsHtv3Q3VXnKTTSit/VSm7QNI6SU9IWirpLQMnkHSupKskXSHpSUl3tEyjiaS1ks6WtELSo5L+RdKvtWw/VtKyMtH9jyT9dsu2g8vxnpR0BfB8vaFI+oikleU835f0qpZtlnSqpFXlXF9RZfey/rqWffeR9LSkV47t4+4NadHvQLut2XbqT2RLOq32nYftZyQtBg4Hlpb3HwL3Dyob3Jo/nmq8qqfL+u3A3wCPA2cA35Y00/Z/lu3HUU00/8dl+3cl/abtZ8v2DwBvp5q+8l+Bvwb+WtLBwEXAHwJLSv1rJR0IGPgu1RzMXy7nuIxqCs0XkXQccE451irgrLL/77XsdizwO8DLy8/+r7avl3R1if9TZb8TgEW2Nw75wfa4tOgjdj6LqJI5VK33H5ZXa9miQXX+3vZm208D2P6/th+xvdX2F4DdgQNb9l9q+6qS2M+nankf2rL9y7bX2d4MfI4qqUI1/tX/tr3Y9jbblwC/KnUPpfo28iXbz9q+iuoPznBOLXGvLN1NfwfMaW3VA+fZfsz2fVTfbAa+1VxK9UT/gD8qZf3Jdu0vqr/UeeXVzdeSun/Pe+UFHAk8DOwF3F/KXg48VMq2AbMG/f+cPegY/x1YSdWifwx4DjiqbDsX+Pag/W8H3l+W1wLvatn2WuDpsnwd8FQ55sDrKao/BCcCtw867mXAZ4f5OVcA/zHoWE8Dv9fyc72mZf+LB45F9UT/A1TfYmZSffPYo+5/u/G+0nUTsfP5MfAK4E+B/wdg+wlJ95ey+23/clAdDyyU/vhPAEcBd9l+TtKjgFr2n9Gy/0uoBja8f6jtwP4t29YBn7P9ucFBS3orME2SXLJxqXvPMD/nwLG+Ncz2Ybm6KH0l1R+Yh4AFtp8c63F6RU8k+unTp+e2veiq/H69wPbTkpYAZ1J1mwy4tZT9+wiH2APYSvWtYJcyVejLB+3zRknvpRrM8C+oul9ua9l+mqQFVK31TwFXlPKvA9dI+nfgJ8BLgSOorhn8uJz3LyT9M1Xf+yFUXS5D+Rrwt5KW2b5L0iuAt9n+9gg/34BLqa4JPMILffV9KX30ETunRcArqZL7gB+WshfdVjnI94HrqaYLvRf4T7afaAjge8D7gUeBPwHe6xcuxEKVRG8A1lC1yD8LYHsJ1beKL5e6q4EPlW3PAO8t65vL8a8eLkjb11BdqL1c0hPAcuAdw+0/RP3FVF02vwH822jr9aKeGNRsxowZTosruunMM8/MoGYTRNK5VH3ffzzM9rXAR22P9M0hOiQt+oiIhht3opc0Q9LN5aGHuySdUcrPlbShPPCwTNI7OxduRESMVTsXY7cCH7d9h6Q9gKWSbizbvmj78+2HF9FbJB0DXEB1+903bJ9Xc0g9x/a5I2yfOTGRxIBxt+htP2D7jrL8JNU9tdM6FVhEr5E0CfgK1QW9g4CTJB1Ub1QRI+tIH72kmcDBwOJSdLqkOyVdJGlKJ84R0QMOoRrlcU25A+RyqsfwI3pa2/fRS3oZ8B3gL8tDF18F/pbqAYu/Bb4AfGSIevOoHndmypT8LYi+MI3tbyNcT/Xk5LCmTp3q/fffv6tBxc7rvvvuY9OmTRppv7YSvaRdqZL8t2xfDWD7oZbtXwcWDFXX9nxgPlS3V7YTR0QvaW3EzJgxg1tvvXWEGhHjc9hhh41qv3buuhFwIbDS9vkt5fu17PYeqocUIppgA9s/uj+9lG3H9nzbc23PnTp16oQFFzGcdlr0b6Z64u1nkpaVsnOoLlDNoeq6WQt8rK0II3rH7cBsSbOoEvyJVKMajsvkyZM7FVc03JYtW9qqP+5Eb/tWth/EaMB14w8nonfZ3irpdKohACYBF9m+q+awIkbUE4OaRfQL29eRxkz0mb5I9BkHJ0aSWbIihpexbiIiGi6JPiKi4ZLoIyIaLok+IqLhkugjIhouiT4iouGS6CMiGi6JPiKi4ZLoIyIaLok+IqLhkugjIhouiT4iouGS6CMiGq4Tc8auBZ4EtgFbbc+VtBdwBTCTavKRE2w/2u65IiJi7DrVov9923Nszy3rZwE32Z4N3FTWIyKiBt3qujkOuKQsXwIc36XzRETECDqR6A3cIGmppHmlbF/bD5TlB4F9O3CeiIgYh07MMHWY7Q2SXgncKOnnrRttW5IHVyp/FOYBTJkypQNhRPSXhQsX1h1C7CTabtHb3lDeNwLXAIcAD0naD6C8bxyi3nzbc23PnTx5crthRETEMNpK9JImS9pjYBl4G7AcuBY4uex2MvC9ds4TERHj127Xzb7ANZIGjnWp7esl3Q5cKekU4F7ghDbPExER49RWore9Bnj9EOWPAEe1c+yIiOiMPBkbEdFwnbjrputuO+aYukOIHvejugMYh1mzZtUdQuwk0qKPiGi4JPqIiIZLoo8YRNIMSTdLWiHpLklnlPK9JN0oaVV5z5N+0ReS6CNebCvwcdsHAYcCp0k6iAzWF32qLy7GRkykMk7TA2X5SUkrgWlUg/UdUXa7BPgB8Mnxnqc8fxLRdX2R6J97zRN1hxA7KUkzgYOBxWSwvuhT6bqJGIaklwHfAf7S9natDdumGrl1qHrzJC2RtGTTpk0TEGnEjiXRRwxB0q5USf5btq8uxSMO1gfbD9g3derUiQk4Ygf6ousmYiKp6jy/EFhp+/yWTQOD9Z1HBwbre8lL0s6KiZFEH/Fibwb+BPiZpGWl7ByqBJ/B+qLvJNFHDGL7VmC4W2I6NljfjBkzOnWoaLinnnqqrfp9keg3v7y9HzIiYmeWTsKIiIYbd4te0oHAFS1FBwCfBvYE/hR4uJSfY/u6cUcYERFtGXeit303MAdA0iRgA9WcsR8Gvmj78x2JMKKhHn/88bpDiD6x6667tlW/U330RwH32L43j3VHjM66devqDiH6xAEHHNBW/U4l+hOBy1rWT5f0QWAJ1eBQj7Zz8M2/9Uw71WNnkAdQI4bV9sVYSbsB7wa+XYq+CryaqlvnAeALw9R7/jHxLVu2tBtGREQMoxN33bwDuMP2QwC2H7K9zfZzwNeBQ4aq1PqY+OTJkzsQRkREDKUTXTcn0dJtI2m/lhH+3gMs78A5IhpnxYoVdYcQfaLWPnpJk4GjgY+1FP+jpDlUI/utHbQtIiImWFuJ3vYWYO9BZX/SVkQRO4kHH3yw7hBiJ9EXQyBc+tz+dYcQPe5tdQcQ0cMyBEJERMP1RYs+oon+6q/+qu4Qok989KMfbat+WvQREQ2XFn1ETZ55Jk98x8RIiz4iouH6okX/zOXn1h1C9Lq3/ajuCCJ6Vlr0ERENl0QfEdFwSfQREQ2XRB8R0XBJ9BERDdcXd90svP7QukOIHnfs286vO4SInpUWfUREwyXRR0Q03KgSvaSLJG2UtLylbC9JN0paVd6nlHJJ+idJqyXdKekN3Qo+IiJGNtoW/cXAMYPKzgJusj0buKmsQzWH7Ozymkc1WXhE35E0SdJPJS0o67MkLS6NmCsk7VZ3jBGjMapEb/sWYPOg4uOAS8ryJcDxLeXfdOU2YE9J+3Ui2IgJdgawsmX9H4Av2n4N8ChwSi1RRYxRO330+7ZMAv4gsG9Zngasa9lvfSmL6BuSpgPvAr5R1gUcCVxVdmlt3ET0tI5cjLVtqsnAR03SPElLJC3ZsmVLJ8KI6KQvAZ8AnivrewOP2d5a1tOAib7RTqJ/aKBLprxvLOUbgBkt+00vZduxPd/2XNtzJ0+e3EYYEZ0l6Vhgo+2l46z/fCNm06ZNHY4uYuzaSfTXAieX5ZOB77WUf7DcfXMo8HhLF09EP3gz8G5Ja4HLqbpsLqC63jTwkOGQDRjYvhEzderUiYg3YodGe3vlZcCPgQMlrZd0CnAecLSkVcAflHWA64A1wGrg68CfdTzqiC6yfbbt6bZnAicCC21/ALgZeF/ZrbVxE9HTRjUEgu2Thtl01BD7GjitnaAietQngcslfRb4KXBhzfFEjEpfjHUTURfbPwB+UJbXAIfUGU/EeGQIhIiIhkuij4houCT6iIiGS6KPiGi4JPqIiIZLoo+IaLgk+oiIhkuij4houCT6iIiGS6KPiGi4JPqIiIZLoo+IaLgk+oiIhkuij4houBETvaSLJG2UtLyl7H9J+rmkOyVdI2nPUj5T0tOSlpXX17oZfEREjGw0LfqLgWMGld0IvM72bwO/AM5u2XaP7TnldWpnwoyIiPEaMdHbvgXYPKjsBttby+ptVPNnRkRED+pEH/1HgH9rWZ8l6aeSFkl6y3CVJM2TtETSki1btnQgjIiIGEpbUwlK+hSwFfhWKXoA2N/2I5LeCHxX0mttPzG4ru35wHyAGTNmuJ04IiJieONu0Uv6EHAs8IEyITi2f2X7kbK8FLgH+M0OxBkRMSYLFy5k4cKFdYfRE8aV6CUdA3wCeLftp1rK95E0qSwfAMwG1nQi0IiIGJ8Ru24kXQYcAUyVtB74DNVdNrsDN0oCuK3cYXM48DeSngWeA061vXnIA0dETID99tuPvffeG4Dly5ePsHczjZjobZ80RPGFw+z7HeA77QYVERGd09bF2IiIXnXkkUcCcOyxx/KmN70J2Hlb9BkCISKi4dKij4hGW7FiBZs379yXCtOij4houLToI6LR1qxZw5o1O/dd3mnRRwxB0p6SriqjtK6U9LuS9pJ0o6RV5X1K3XFGjEYSfcTQLgCut/1bwOuBlcBZwE22ZwM3lfWInpdEHzGIpFdQPfx3IYDtZ2w/BhwHXFJ2uwQ4vp4II8YmiT7ixWYBDwP/UkZi/YakycC+th8o+zwI7DtU5daRWTdt2jRBIUcML4k+4sV2Ad4AfNX2wcAWBnXTlIH8hhx11fZ823Ntz506dWrXg40YSRJ9xIutB9bbXlzWr6JK/A9J2g+gvG+sKb6IMUmijxjE9oPAOkkHlqKjgBXAtcDJpexk4Hs1hBcxZrmPPmJofw58S9JuVENtf5iqYXSlpFOAe4ETaowvYtSS6COGYHsZMHeITUdNdCwR7Rqx60bSRZI2SlreUnaupA2SlpXXO1u2nS1ptaS7Jb29W4FHRMTojKaP/mLgmCHKv2h7TnldByDpIOBE4LWlzj8PzDgVERH1GDHR274FGO3Qb8cBl5e5Y38JrAYOaSO+iIhoUzt33Zwu6c7StTMw5sc0YF3LPutLWURE1GS8if6rwKuBOcADwBfGeoDWpwe3bNkyzjAiImIk40r0th+yvc32c8DXeaF7ZgMwo2XX6aVsqGM8//Tg5MmTxxNGRESMwrgS/cDTgcV7gIE7cq4FTpS0u6RZwGzgJ+2FGBER7RjxPnpJlwFHAFMlrQc+AxwhaQ7VWB9rgY8B2L5L0pVUTxFuBU6zva07oUdExGiMmOhtnzRE8YU72P9zwOfaCSoiIjonY91ERDRcEn1ERMMl0UdENFwSfUREwyXRR0Q0XBJ9RETDJdFHRDRcEn1ERMMl0UdENFwSfUREwyXRR0Q0XBJ9RETDJdFHRDRcEn1ERMMl0UdENNxoJh65CDgW2Gj7daXsCuDAssuewGO250iaCawE7i7bbrN9aqeDjugXzz77LPfff3/dYUQHLVy4cFz1zj//fADOPPPMMdddvHjxkOWjnW97xEQPXAx8GfjmQIHt9w8sS/oC8HjL/vfYnjOqs0dERNeNZoapW0pL/UUkCTgBOLKzYUVERKe020f/FuAh26taymZJ+qmkRZLe0ubxI2oh6b9JukvSckmXSfo1SbMkLZa0WtIVknarO86I0RhN182OnARc1rL+ALC/7UckvRH4rqTX2n5icEVJ84B5AFOmTGkzjIjOkTQN+AvgINtPlwnvTwTeCXzR9uWSvgacAny1xlCjBkce2V4HxoIFC8ZcZ7zXBQaMu0UvaRfgvcAVA2W2f2X7kbK8FLgH+M2h6tueb3uu7bmTJ08ebxgR3bIL8Ovl9/ylVI2YI4GryvZLgONrii1iTNpp0f8B8HPb6wcKJO0DbLa9TdIBwGxgTZsxRkwo2xskfR64D3gauAFYSnV32day23pg2kjH+sUvfsHRRx/dtVhj59Dut4gRW/SSLgN+DBwoab2kU8qmE9m+2wbgcOBOScuoWj6n2t7cVoQRE0zSFOA4YBbwG8Bk4Jgx1J8naYmkJdu2betSlBGjN5q7bk4apvxDQ5R9B/hO+2FF1OoPgF/afhhA0tXAm4E9Je1SWvXTgQ1DVbY9H5hf6nrt2rUTEnTEcPJkbMSL3QccKuml5Rbio4AVwM3A+8o+JwPfqym+iDFJoo8YxPZiqq7HO4CfUf0/mQ98EjhT0mpgb+DC2oKMGIN2b6+MaCTbnwE+M6h4DXBIDeFEtCUt+oiIhkuLPqK7NgFbynuvmEriGUmvxTRcPK8aTeUk+ogusr2PpCW259Ydy4DEM7Jei6ndeNJ1ExHRcEn0EREN1xNdN49Peo4Fe/5H3WHslG47ZtQPfL7Ioddf38FI2vN7N9xQdwg7Mr/uAAZJPCPrtZjaiict+oguK0/K9ozEM7Jei6ndeJLoIyIaLok+okskHSPp7jJRyVk1xTBD0s2SVpSJVM4o5XtJulHSqvI+oZNCSJpUJihaUNZrm9RF0p6SrpL0c0krJf1unZ9PNya96Yk++qhPL/WzN4mkScBXgKOphjS+XdK1tldMcChbgY/bvkPSHsBSSTcCHwJusn1e+SN0FtUQDxPlDGAl8PKy/g/UN6nLBcD1tt9XEuhLgXOo4fPp1qQ3adFHdMchwGrba2w/A1xONfTxhLL9gO07yvKTVMl1WonlkrLbhE6iImk68C7gG2Vd1DSpi6RXUA2vfiGA7WdsP0aNnw9dmPQmLfpohB78ZjINWNeyvh54U02xACBpJnAwsBjY1/YDZdODwL4TGMqXgE8Ae5T1vRnHpC4dMgt4GPgXSa+nmmDmDGr6fDo56U2rERO9pBnAN6l+UAPzbV8gaS+qaQRnAmuBE2w/Wv46X0D1VeMp4EMDLYrhPHnv/Sz8yKfHEndEjIGkl1HNFfGXtp+o/ptWbFuSJyiOY4GNtpdKOmIizjmCXYA3AH9ue7GkC6i6aZ43wZ9P66Q3jwHfZgyT3gxnNF03A318BwGHAqdJOojqw7jJ9mzgJl74cN5BNYXgbKrJvzN5cuyMNgAzWtaHnaik2yTtSpXkv2X76lL8kKT9yvb9gI0TFM6bgXdLWkvVnXUkVcNwz9JVARP7Wa0H1pehqaHqHnkD9X0+z096Y/tZYLtJb8o+Y/58Rkz04+jjOw74piu3lQD3G0tQEQ1wOzC73C2xG9UFtWsnOojyDftCYKXt81s2XUs1eQpM4CQqts+2Pd32TKrPZKHtD1DTpC62HwTWSTqwFA1MMlPL50O3Jr2xPeoXVTfNfVRXyh9rKdfAOrAAOKxl203A3BGO67zy6vJryVh+1zvxouq+/AVwD/CpiT5/ieGw8vPfCSwrr3dS9YvfBKwC/h3Yq4bYjgAWlOUDgJ8Aq6m6K3afwDjmAEvKZ/RdYEqdnw/wP4GfA8uB/wPs3u7no3LgEZU+vkXA52xfLekx23u2bH/U9pRyX+x5tm8t5TcBn7S9ZNDx5lF17QC8cVRBRIzfUvfQaIQRE2lUt1eOsY9vVH2Ttufbnpv/fBER3TVioh9HH9+1wAdVORR43C/cphQRERNsxK4bSYcBP6SaJPm5UnwO1b24VwL7A/dS3V65ufxh+DLVLUFPAR8e3G0zxDlG138UMX7puomd1qj76LsaRBJ9dF8Sfey0MgRCRETDJdFHRDRcEn1ERMMl0UdENFyvjF65CdhS3vvRVPo3dujv+Ecb+6u6HUhEr+qJu24AJC3p17si+jl26O/4+zn2iImSrpuIiIZLoo+IaLheSvTz6w6gDf0cO/R3/P0ce8SE6Jk++oiI6I5eatFHREQX1J7oJR0j6W5JqyWdNXKN+klaK+lnkpZJWlLK9pJ0o6RV5X1K3XEOkHSRpI2SlreUDRlvGXX0n8q/x52S3lBf5MPGfq6kDeXzXybpnS3bzi6x3y3p7fVEHdFbak30kiYBX6GaZ/Yg4KQyH20/+H3bc1pu7RtuDt1ecDEvnmC4X+b8vZihJ0f+Yvn859i+DqD87pwIvLbU+efyOxaxU6u7RX8IsNr2GtvPUE0WfFzNMY3XcHPo1s72LcDmQcV9MefvMLEP5zjgctu/sv1LqmnXDulacBF9ou5EPw1Y17K+vpT1OgM3SFpapkQE2LdlgpUHgX3rCW3Uhou3X/5NTi9dSxe1dJP1S+wRE6ruRN+vDrP9BqpujtMkHd660dWtTH1zO1O/xUvVnfRqqkmdHwC+UG84Eb2t7kQ/qvlle43tDeV9I3ANVffAcHPo9qq25vytk+2HbG+z/RzwdV7onun52CPqUHeivx2YLWmWpN2oLqRdW3NMOyRpsqQ9BpaBtwHLGX4O3V7Vt3P+Drpm8B6qzx+q2E+UtLukWVQXlH8y0fFF9JpaR6+0vVXS6cD3gUnARbbvqjOmUdgXuKaaGpddgEttXy/pduBKSadQ5tCtMcbtSLoMOAKYKmk98BngPIaO9zrgnVQXMp8CPjzhAbcYJvYjJM2h6m5aC3wMwPZdkq4EVgBbgdNsb6sj7ohekidjIyIaru6um4iI6LIk+oiIhkuij4houCT6iIiGS6KPiGi4JPqIiIZLoo+IaLgk+oiIhvv/JX1SHcALJOsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize raw and wrapped environment:\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.imshow(ob_unwrap)\n",
    "plt.title('Raw env')\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.imshow(ob.squeeze(), cmap='gray')\n",
    "plt.title('Wrapped env')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "378d0f15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Discrete(4)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.action_space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6dc2b8a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "acs_dim = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41fb8949",
   "metadata": {},
   "source": [
    "#  Set Up Exploration\n",
    "\n",
    "Categorical DQN uses $\\epsilon$-greedy exploration strategy which means that with $\\epsilon$ probability random action is taken instead of Q-function maximizing one. $\\epsilon$ is typically annealed over the course of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fae09e70",
   "metadata": {},
   "outputs": [],
   "source": [
    "exploration = EpsilonGreedy(eps=eps_schedule)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8519087e",
   "metadata": {},
   "source": [
    "# Define ArgmaxQValue actor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d2318887",
   "metadata": {},
   "outputs": [],
   "source": [
    "actor = ArgmaxQValue(\n",
    "        exploration=exploration\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3eb21f0",
   "metadata": {},
   "source": [
    "# Define CategoricalDQN critic:\n",
    "\n",
    "!Note: Atari 2600 environments often have some blinking visual elements, that is why observation may miss some details due to the blinking period, therefore introducing *partial observability*. To counteract this, critic network receives a stack of a current frame and 3 most recent lags. By doing so, critic, in theory, should be able to reconstruct actual state from frames history.\n",
    "\n",
    "ReLAx offers a simple interface to use lagged observations by specifying the following args:\n",
    "\n",
    "`obs_nlags`: int - specify the number of lags in addition to current observation\n",
    "\n",
    "`obs_expand_axis`: int or None - if not None, creates an extra dimension in observation array to concat along, if needed\n",
    "\n",
    "`obs_concat_axis`: int - axis to concatenate lags along\n",
    "\n",
    "`obs_padding`: str - ('zeros', 'first') value to pad starting observations with.\n",
    "\n",
    "Observation shape of the observations returned by DeepMind environment wrapper is (84, 84, 1) so, by specifying:\n",
    "\n",
    "`obs_nlags`=3\n",
    "\n",
    "`obs_expand_axis`=None\n",
    "\n",
    "`obs_concat_axis`=-1\n",
    "\n",
    "`obs_padding=`'zeros'\n",
    "\n",
    "We end up with a stacked lagged observation with a shape (84, 84, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5f9ed791",
   "metadata": {},
   "outputs": [],
   "source": [
    "critic = CategoricalDQN( # Categorical DQN\n",
    "    critic_net=AtariDistributionalQCNN( # Categorical DQN\n",
    "        in_channels=4, # 1 current frame + 3 recent lags\n",
    "        acs_dim=acs_dim,\n",
    "        n_atoms=51,\n",
    "        v_min=-10,\n",
    "        v_max=10\n",
    "    ),\n",
    "    device=torch.device('cuda'),\n",
    "    learning_rate=lr_schedule,\n",
    "    batch_size=32,\n",
    "    gamma=0.99,\n",
    "    target_updates_freq=10000,\n",
    "    n_steps_per_update=1,\n",
    "    update_freq=4,\n",
    "    double_q=False,\n",
    "    obs_nlags=3,\n",
    "    obs_expand_axis=None,\n",
    "    obs_concat_axis=-1,\n",
    "    obs_padding='zeros'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9896a2ef",
   "metadata": {},
   "source": [
    "# Provide actor with critic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "28fabc7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "actor.set_critic(critic)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0073e359",
   "metadata": {},
   "source": [
    "# Wrap envs into Sampler:\n",
    "\n",
    "!Note: As we are using lags as an input to critic (and actor is using critic to select actions), our sampler also should accumulate lags to ensure correct input on a sampling phase. To make Sampler use lags, you should provide it with the same set of args:\n",
    "\n",
    "`obs_nlags`: int - specify the number of lags in addition to current observation\n",
    "\n",
    "`obs_expand_axis`: int or None - if not None, creates an extra dimension in observation array to concat along, if needed\n",
    "\n",
    "`obs_concat_axis`: int - axis to concatenate lags along\n",
    "\n",
    "`obs_padding`: str - ('zeros', 'first') value to pad starting observations with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4ca4c36f",
   "metadata": {},
   "outputs": [],
   "source": [
    "sampler = Sampler(env=env,\n",
    "                  obs_nlags=3,\n",
    "                  obs_expand_axis=None,\n",
    "                  obs_concat_axis=-1,\n",
    "                  obs_padding='zeros')\n",
    "\n",
    "eval_sampler = Sampler(env=eval_env,\n",
    "                       obs_nlags=3,\n",
    "                       obs_expand_axis=None,\n",
    "                       obs_concat_axis=-1,\n",
    "                       obs_padding='zeros')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1628bb9",
   "metadata": {},
   "source": [
    "# Create replay buffer\n",
    "\n",
    "!Note: ReplayBuffer is designed not to store lags explicitly (and not to store next states also) that is why the number of lags will not affect the amount of RAM used. However, as it still has to store 1m experience history and each observation is an array of shape (84, 84, 1) with a dtype uint8, the total amount of RAM usage by ReplayBuffer $ \\approx $ 10^6 \\* 84 \\* 84 bytes ~= 7 gigabytes. In total, DQN training requires $ \\approx $ 9-10 gigabytes of RAM. If your machine does not have enough RAM - consider decreasing buffer size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "38793b37",
   "metadata": {},
   "outputs": [],
   "source": [
    "replay_buffer = ReplayBuffer(int(1e+6))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a363bcb",
   "metadata": {},
   "source": [
    "# Visualize untrained actor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1ba08d72",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Simulation done, total reward: 0.0\n",
      "Loading video...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" autoplay \n",
       "                    loop controls style=\"height: 400px;\">\n",
       "                    <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAABphtZGF0AAACrgYF//+q3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE1NSByMjkxNyAwYTg0ZDk4IC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxOCAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD0tMiB0aHJlYWRzPTcgbG9va2FoZWFkX3RocmVhZHM9MSBzbGljZWRfdGhyZWFkcz0wIG5yPTAgZGVjaW1hdGU9MSBpbnRlcmxhY2VkPTAgYmx1cmF5X2NvbXBhdD0wIGNvbnN0cmFpbmVkX2ludHJhPTAgYmZyYW1lcz0zIGJfcHlyYW1pZD0yIGJfYWRhcHQ9MSBiX2JpYXM9MCBkaXJlY3Q9MSB3ZWlnaHRiPTEgb3Blbl9nb3A9MCB3ZWlnaHRwPTIga2V5aW50PTI1MCBrZXlpbnRfbWluPTI1IHNjZW5lY3V0PTQwIGludHJhX3JlZnJlc2g9MCByY19sb29rYWhlYWQ9NDAgcmM9Y3JmIG1idHJlZT0xIGNyZj0yMy4wIHFjb21wPTAuNjAgcXBtaW49MCBxcG1heD02OSBxcHN0ZXA9NCBpcF9yYXRpbz0xLjQwIGFxPTE6MS4wMACAAAABxmWIhAAz//727L4FNhTIUGV5w7TCGgBbbRnuhdMtyDBPyB0r1rd4/5Qp55zwiSeu7vzfFf+2f8P2JvdKcCbU3w1XD2CeN17jUDBJ6BF9pl0HsdLgUSGY8qkywaUaZR2uVgU9lfIcWco6inY/5yAhFwH+6eDF91Rgw+loSI68dVgKdF8aXvoGuKIMNUuPcL3rKRMj2xIb1wP7huTzIDTIOXKrOk0iM6pS6NB3nlDBZ/1jjUJV3JI5Dd+LXLqB5z+EoDWPyL/1HNzLgaqKAlH2U8j48QlggBiclCNEvkl0Jo8X0WDfrjd2/u2e04UkIojucL2lbX+XkKtHYAKnmrbrnuYjpV71oa1gafajmrge56JyK4K0VRmoNpxGJijOtc45y5invC3AP18GvO+srbJKJNO9zHgTro/0OfJnR+Fupbqn4SoB4Xf8fpbaSvrJzVnUgoDfKS+bFOQSl0sg+Mlya+eYKBitYD2uzxRMu2SnwADWVSg8tsaFxzMCjG7UZijPV2aAcGnitENediFSJHdfeXuoy8r5LyXMS6RiFoE0JWkRPMdYeUoqwnwH3VIfnrVjcH1kTQz91biqcdjZJAiKz9Td4ydbF4EAAACSQZokbEL//oywC50/nACLw0WAa3CYV3lGuv63Hb68iuhlzF7M4QfuruFSMJF6+X/4PeZ0pbBjewfkBv5g1BvPqKOxgPP2ury6V8cvZVVSGrDbDI6pphYU82iwdIBsXmTU0VEB9Qin8SrTJrdhFaYPzxVjA/5bwGPcS2INF4A7dBFFskZYGWZyjJtQIpL5n/4Q4IgAAAB9QZ5CeIV/AVpqxidJ//PbhbT7ndKhsixhHanBd82wA1U/Y9pBEFoGp7mBTLEC5PzZU65v2Ikof1FQqnSMP5TA4WrJcnpoXmgv/LBDF3G4cy2bwALC6iXgUS2hg4XP3HKfy9n4jzhYgFnVhWNfMtQOyRZ7G34M+y/7bsrdNu8AAABrAZ5hdEJ/AaPye8Z+zqD0SajLSzWcAG0n2VKkY3Gw51GPg6dWRsn77/SyKnIfk8LgmU/RH1bAsMr9HxH8eyd4Td51ZV1palmEkZ759rthA6Zd1X9o/k+WDCnga671JO4TE8VBbb5tx7LcKXAAAAAnAZ5jakJ/AH8C2KAlTU/D5/eEAIpqPpobMUpmgLNIF8dy7ZsqCreBAAAAMkGaZ0moQWiZTAhP//3XtSgAnTP/+Y3Rrmzu6GP1XrZ8qUK+4TBeimgFC5PqMtmScELhAAAAGkGehUURLCv/OarghLvOpsT8acD8vGyfY4t/AAAACwGepmpCfz/QaYi5AAADa21vb3YAAABsbXZoZAAAAAAAAAAAAAAAAAAAA+gAAAELAAEAAAEAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAKVdHJhawAAAFx0a2hkAAAAAwAAAAAAAAAAAAAAAQAAAAAAAAELAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAACgAAAA0gAAAAAAJGVkdHMAAAAcZWxzdAAAAAAAAAABAAABCwAABAAAAQAAAAACDW1kaWEAAAAgbWRoZAAAAAAAAAAAAAAAAAAAPAAAABAAVcQAAAAAAC1oZGxyAAAAAAAAAAB2aWRlAAAAAAAAAAAAAAAAVmlkZW9IYW5kbGVyAAAAAbhtaW5mAAAAFHZtaGQAAAABAAAAAAAAAAAAAAAkZGluZgAAABxkcmVmAAAAAAAAAAEAAAAMdXJsIAAAAAEAAAF4c3RibAAAAJhzdHNkAAAAAAAAAAEAAACIYXZjMQAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAACgANIASAAAAEgAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABj//wAAADJhdmNDAWQADP/hABlnZAAMrNlCh34iEAAAAwAQAAADA8DxQplgAQAGaOvjyyLAAAAAGHN0dHMAAAAAAAAAAQAAAAgAAAIAAAAAFHN0c3MAAAAAAAAAAQAAAAEAAABIY3R0cwAAAAAAAAAHAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAgAAAAAAgAAAgAAAAAcc3RzYwAAAAAAAAABAAAAAQAAAAgAAAABAAAANHN0c3oAAAAAAAAAAAAAAAgAAAR8AAAAlgAAAIEAAABvAAAAKwAAADYAAAAeAAAADwAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1OC4yOS4xMDA=\" type=\"video/mp4\" />\n",
       "                 </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_actor(env=eval_env, actor=actor, nsteps=1000, train_sampling=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5521a071",
   "metadata": {},
   "source": [
    "# Run Training and Validation loop:\n",
    "\n",
    "!Note: we have constructed our exploration schedule to be 30m timestamps long, however, for the time-saving purpose, we will run demonstration training only for 3m steps. In papers, DQN and its augmentations are trained for 200m steps, which may require several days of learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "cf675b2b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3000000/3000000 [6:48:39<00:00, 122.35it/s]  \n"
     ]
    }
   ],
   "source": [
    "log_every = 50000\n",
    "n_steps = 3000000\n",
    "logdir = '/home/nikita/Desktop/RelaxExamples/Categorical_DQN/tensorboard_logs'\n",
    "run_name = f'categorical_dqn_{env_name}'\n",
    "logger = SummaryWriter(log_dir=f'{logdir}/{run_name}')\n",
    "all_logs = []\n",
    "\n",
    "for step in tqdm(range(n_steps)): # \n",
    "    \n",
    "    # sample training batch\n",
    "    unit_pathlist = sampler.sample(actor=actor,\n",
    "                                   n_transitions=1,\n",
    "                                   max_path_length=None, \n",
    "                                   reset_when_not_done=False,\n",
    "                                   train_sampling=True)\n",
    "    \n",
    "    # add transitions to replay buffer\n",
    "    replay_buffer.add_paths(unit_pathlist)\n",
    "    \n",
    "    # update critic\n",
    "    critic_logs = critic.update(replay_buffer)\n",
    "        \n",
    "    # update actor\n",
    "    actor_logs = actor.update(replay_buffer)\n",
    "    \n",
    "    if step % log_every == 0 and step > 0:\n",
    "        \n",
    "        # add unwrapped reward logs\n",
    "        mean_unwr_train = np.mean(get_wrapper_by_name(env, 'Monitor').get_episode_rewards()[-20:])\n",
    "        unwr_rews = {'average_train_unwr_rews': mean_unwr_train}\n",
    "        \n",
    "        # create logs of buffer\n",
    "        buffer_logs = replay_buffer.create_logs(prefix='buffer')\n",
    "\n",
    "        logs = {**critic_logs,\n",
    "                **actor_logs,\n",
    "                **buffer_logs,\n",
    "                **unwr_rews}\n",
    "        all_logs.append(logs)\n",
    "        \n",
    "        for key, value in logs.items():\n",
    "            logger.add_scalar(key, value, step)\n",
    "    \n",
    "logger.flush()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d3b0dee7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "      <iframe id=\"tensorboard-frame-5f57613b27d65135\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
       "      </iframe>\n",
       "      <script>\n",
       "        (function() {\n",
       "          const frame = document.getElementById(\"tensorboard-frame-5f57613b27d65135\");\n",
       "          const url = new URL(\"/\", window.location);\n",
       "          const port = 6008;\n",
       "          if (port) {\n",
       "            url.port = port;\n",
       "          }\n",
       "          frame.src = url;\n",
       "        })();\n",
       "      </script>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%load_ext tensorboard\n",
    "%tensorboard --logdir /home/nikita/Desktop/RelaxExamples/Categorical_DQN/tensorboard_logs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c5ed3c",
   "metadata": {},
   "source": [
    "# Plot learning curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d36c25e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8VfX9+PHXO5sMEsgEQggzyN4gSgEnWrVqq3W0bq1+a9fXLttvW7+2319bW7utra1Vq3W0dWsdqCDK3giBhIQkZJC997if3x/nBELIuEnuSO59Px+PPEjOPfec9+cmnPf5jPP5iDEGpZRS/ivA2wEopZTyLk0ESinl5zQRKKWUn9NEoJRSfk4TgVJK+TlNBEop5ec0ESiXE5G3RORmb8fhKSKSKCKbRKRORB72djwDJSJrRKTA23Eo79FE4ENEJFdELvB2HMaYS4wxT3k7Dg+6CygHRhtj7hvqwUTkFhHpEJF6++uYiNwz9DA9yy7Hx0M8xgIR2S0ijfa/C1wVnzpFE4EaEBEJ8nYMQ+WGMkwC0s0gns7sI5atxphIY0wk8FngIRFZOMBjjGgiEgK8CjwDjAGeAl61tysX0kTgJ0TkMhHZJyLVIrJFROZ1ee27IpJtN22ki8hVXV67RUQ2i8ivRaQCeKDzTk9EfikiVSKSIyKXdHnPRhG5o8v7+9p3cpdmlfdE5BEReaaPcnzGLketHfM6e/tptSEReaDzOCKSKiJGRG4XkePAB3bz1b3djr1fRK62v58pIutFpFJEMkTk2l7ieRK4Gfi2ffd+gYiEishvRKTI/vqNiITa+68RkQIR+Y6IFANP9Pe7M8bsBQ4DZ/VWHnv7Cvt3W22XZU2XOG8VkcP253xMRL7Ux2f8VfvvINn++U4RybI/i9dEZHy3OIK6vHejiNwhImcBfwLOtj+X6l7OdVhELuvyc5CIlInIImANEAT8xhjTYoz5HSDAef19ZmpgNBH4AftO8m/Al4BY4M/Aa50XJyAbWAVEA/8LPCMi47ocYjlwDEgE/q/LtgwgDngIeFxEpJcQ+tr3WWCHHdcDwBf7KMcy4O/At4AY4FNAbn/l72I11sX0YuA54Poux56FdWf/pohEAOvt2BKA64A/2vucxhhzC/AP4CH7Dv494PvACmABMB9YBvxPl7clAWPt893VX9AishSYAezqrTwiMgF4E/iJfexvAi+KSLy9bylwGTAauBX4tX2x7X6uHwK3AKuNMQUich7wU+BaYByQBzzfX8zGmMPA3Zyq2cT0sutpvwes3025MWYPMBs40K2mdcDerlxIE4F/uAv4szFmuzGmw26/b8G6WGGM+ZcxpsgY4zDGvAAcxbp4dSoyxvzeGNNujGmyt+UZY/5ijOnAqrKPw0oUPelxXxFJAZYCPzTGtBpjPgZe66MctwN/M8ast2MtNMYcGcDn8IAxpsEuw8vAAhGZZL92I/CSMaYF64KZa4x5wi7zXuBF4Bonz3Mj8KAxptQYU4aVXLsmOAfwI/sut6nHI8AK+86+DitRPo31e+mtPF8A/mOM+Y/92azHShyXAhhj3jTGZBvLh8C7WMm/k4jIr4CLgLV23J1l+ZsxZo/92dyPdZef6uRn0Z9ngStEJNz++Qas5AAQCdR0278GiHLRuZVNE4F/mATcZ19Yqu1q+kSgs4p/k5xqNqoG5mDdvXfK7+GYxZ3fGGMa7W8jezl/b/uOByq7bOvtXJ0mYtVeBuvksY0xdVh30NfZm67HurMH6/Na3u3zuhHrTt4Z47HunDvl2ds6lRljmvs5xjZjTIwxJso+72zg//VWHjvma7rFfC5W0kVELhGRbXbzTjVWguj6O47BumH4qTGm68X3tLIYY+qBCmBCP/GfQURS5FQHeL19vCysZq/L7WRwBVZyAKjHqsF0NRqoG+i5Vd80EfiHfOD/7AtL51e4MeY5+474L8C9QKxdhT+I1RbbyV1T1J4Axna5GwTrYt+bfGBqL681AF2P09NFu3s5ngOuF5GzgTBgQ5fzfNjt84o0xjg7cqcI68LcKcXe1lscfTLGlGDVSC7v/lKX7/OBp7vFHGGM+ZndBPgi8Esg0f4d/4fTf8dVWDWhJ0TknN7KYjebxQKFWJ859P65n1ZOY8zxzg5wuxO8U2fz0GewOt2z7O2HgHndmhzn2duVC2ki8D3BIhLW5SsI60J/t4gsF0uEiHxaRKKACKz/sGVgdSpi1QjczhiTh9V88YCIhNgX5O4Xu64eB24VkfNFJEBEJojITPu1fcB1IhIsIkuAzzkRwn+wLnIPAi8YYxz29jeAGSLyRft4wSKy1O4AdcZzwP+ISLyIxAE/xBr5MigiEgtcRd8XwGew7qovFpFA+3e/xu7wDQFCsX7H7WJ11l/U/QDGmI3YTWR2f0xnWW4VaxhnKFatZLsxJtduPioEvmCf8zZOT9QlQLL0P8rneTueezhVGwDYCHQAXxWrA76zc/+Dfo6nBkgTge/5D9DU5esBY8wu4E7gD1h3fllYHYIYY9KBh4GtWP9x5wKbPRjvjcDZWM0NPwFewOq/OIMxZgd2RydWW/GHnLpb/QHWRagKq03+2Z6O0e14LcBLwAVd97ebjS7CajYqwmra+jnWxdQZP8FKcAeAT4A99raB6BxtU4/VdFIGfKWPsuRj3VF/z943H6tTPcAuz1eBf2J9PjfQS1+M3bdwG/C6iCyyO79/gFWjOIH1GV/X5S132uepwGq+2tLltQ+wklexiJT3EfsJrL+/lVi//87trcCVwE1AtR3XlfZ25UKiC9Oo4UREXgCOGGN+5O1YlPIXWiNQXmU3uUy1m3rWYd3VvuLtuJTyJz75RKIaUZKwmmdigQLgHnu4plLKQ7RpSCml/Jw2DSmllJ8bEU1DcXFxJjU11dthKKXUiLJ79+5yY0x8f/uNiESQmprKrl3dp1lRSinVFxHJ638vbRpSSim/p4lAKaX8nCYCpZTyc5oIlFLKz2kiUEopP6eJQCml/JwmAqWU8nOaCJRSw0JFfQuv7iv0dhh+SROBUmpYeOyjY3zt+X3kVzb2v7NyKU0ESqlhYXOWtXbNgYLu69Urd9NEoJTyuqqGVg4V1QJwoLDay9H4H00ESimv23qsAmMgPCSQT7RG4HGaCJRSXrc5q5yIkEAunzeeTwprcDh0nRRP0kSglPK6zVnlLJ8Sy6JJMdQ1t5OnHcYepYlAKeVVhdVN5FY0cs60OOYlxwBwoED7CTxJE4FSyqs6RwudMy2W6QmRhAYFaD+Bh42IhWmUUr5rc1Y5cZEhpCVGISLMHj9ah5B6mNYIlFJeY4xhc1YFK6fGISIAzEuO4WBRDR3aYewxmgiUUl6TWVJPeX0L506LO7lt7oRoGls7OFZW78XI/IsmAqWU13T2D6ycFnty27zkaECfMPYkTQRKKa/ZnFVOamw4yWPCT26bEh9pPVhWqInAUzQRKKW8oq3DwfacSlZ2aRYCCAwQ5oyP1iGkHqSJQCnlFQcKqqlvaT+tf6DT3ORoDhXV0t7h8EJk/kcTgVLKKzZnVSACZ0+JPeO1ecnRtLQ7OFqqHcaeoIlAKeUVH2eVM3v8aMZEhJzxWucTxvpgmWdoIlBKeVxjazt7j1dxztQzm4UAJo0NJyosiP3aT+ARmgiUUh63I6eStg7DOT30DwAEBAhzJ0TryCEP0USglPK4LdkVhAQGsDR1bK/7zE2O5vCJWlraOzwYmX/SRKCU8rjdeVXMS45mVEhgr/vMmxBDW4chs1g7jN1NE4FSyqM6HIb0olrmTIjuc7+TTxjr0pVup4lAKeVROeUNNLV19JsIkseMIiY8WEcOeYAmAqWURx0qsi7ss8eP7nM/EavDWOcccj9NBEopjzpUVEtIUADTEiL73Xd+cgyZJXU0t2mHsTu5bWEaEZkI/B1IBAzwmDHmtyIyFngBSAVygWuNMVXuikMpNbwcLKxhZlIUwYH934fOTY6m3WFIP1HLopQxHojudBX1LaSfqD1je0JUGGlJUR6Px13cuUJZO3CfMWaPiEQBu0VkPXAL8L4x5mci8l3gu8B33BiHUmqYMMZwqKiWS+cmObV/Z/NRepF3EsE9/9jDjpzKM7YHBwo7v38BMeFnPhU9ErktERhjTgAn7O/rROQwMAH4DLDG3u0pYCOaCJTyCwVVTdQ0tTF7fN8dxZ0mxIwiKjSIjOI6N0d2puMVjezIqeTWc1L59NxxJ7dnlNTx/ZcPsud4FefNTPR4XO7gkTWLRSQVWAhsBxLtJAFQjNV01NN77gLuAkhJSXF/kEqp02SW1PH6/iLeOHCCWeNG88iNi4Z8zENFVjNLfx3FnUSEGUlRXkkEr+wrBOCOVVOYEDPq5PbZ46P50auH2J2nicBpIhIJvAh83RhT27kuKYAxxohIjwuTGmMeAx4DWLJkiS5eqpQHnKhp4sXdBby+/wQZJXUECMSEh7ApswxjDF3//w7GoaIaAgOEs8Y5lwgA0pKieGN/kUvO7yxjDC/vLWTFlLGnJQGAUSGBzB4/mt15vtO16dZRQyISjJUE/mGMecneXCIi4+zXxwGl7oxBKeUch8PwuUe38st3M4kKC+LBz8xm+/cu4N6106hraaeioXXI5zhUVMvU+AjCgnt/ori7mUlR1Da3U1zbPOTzO2tffjU55Q1cvTC5x9cXTRrDvvxq2nxkvQS3JQKxUvfjwGFjzK+6vPQacLP9/c3Aq+6KQSnlvKOl9RRWN/HTq+fy73tWctPZqcRHhTI5LgKA3PKGIZ/jYGENc5zsH+iUlmiNzjniweahV/YWEhoUwLpeOrUXTxpDc5uDwz2MKBqJ3FkjOAf4InCeiOyzvy4FfgZcKCJHgQvsn5VSXrYz1xod031q6FQ7ERwbYiIorWumtK6FWU72D3SamWTt76l+grYOB68fOMEFsxIZHRbc4z6LJ1kjmHylecido4Y+Bnpr0DvfXedVSg3OztxKEqJCmTj29Dbx5DGjCAyQIdcIOjuK+5taorvo8GCSRod5LBF8mFFGZUMrVy+c0Os+46JHMT46jN15Vdx6zmSPxOVO+mSxUgqAnTmVLJ089owO2eDAAFLGhpNbMbREkG4ngoHWCMDqMPZUInh5byGxESF8akZ8n/stTh3rMzUCTQRKKQqrmyiqaWbppJ4f2kqNDSenvHFI5zhYWMOk2PBem1v6MjMpiqyyercvZl/T1Mb6wyVcPn98v08+L06J4URNM0XVTW6NyRM0ESil2Gk/Pbt0cs8LxaTGRZBX0YAxgx/Jfaio1unnB7pLS4qitd0x5FpJf94+eILWdgdX9tEs1GnxJOuz6q1W4HAYvvzsHh7ZkDWkz80TNBEopdiRW0lUaNDJjtnuJsdF0NjaQWldy6COX9PUxvHKRqefKO6uc14fd48cemlPIVPiIpif3H+cM8dFMSo4sNdE8FFWOW8eOMEv3sngvn/tp7V9+A411USglGJXbiWLJo0hMKDn8R2psdbIoZxBdhinD/CJ4u6mJUQSGCBu7ScoqGpke04lVy2c4NSDa8GBAcyfGN1rInh6ax5xkSF87fzpvLSnkJv+tp2axjZXh+0SmgiU8nNVDa1kltSzrJdmIWDIzxKcWoNgcDWC0KBAJsdFuLVG8Oq+IgCnmoU6LZk0lvQTtTS2tp+2vaCqkQ+OlHDd0hS+ceEMfvP5BezJq+aqRzdzvGJofS3uoIlAKT+3y76jXdJLRzHA+JhRhAQGDLpGcKiolsTRocRHhQ7q/eD+kUOv7C1kWepYJo4Nd/o9iyeNocNh2J9/+uI5z24/DsD1y6150q5cOIGnb19GZUMrV/1xM3uPD6/RRpoIlPJzu3IrCQkMYP7EmF73CQwQJo4dNehEMJgnirubmRjF8cpGGlra+995gIqqmzhaWs/Fc5ybHrvTwhTrM9vT5cLe0t7BCzvzOf+sxNPmKVo+JZaX7lnJqJBA7vvn/mHVgayJQCk/tyO3knnJ0f3O/zM5LnJQo3aaWjvILqsfdP9Ap84O48wS19cKtudUALBiSu/NYz2JCQ9hWkIku3JPrVnw9sFiKhpa+eKKSWfsPyU+knvWTOVYeQMZbijHYGkiUMqPNbV2cLCwhiWp/V8AJ8eFk1fRiMMxsDvZw8W1OAzMHuATxd25c6qJ7ccqGR3W+6ipviyZNIY9x6tPfi5Pb80jNTacc6fF9bj/RbOSELESxnChiUApP2bNoGlYNrn/1b9S4yJoaXdwYoCzgA50DYLeJI8ZRXhIoFs6jLfnVLJs8theR031ZdGkMdQ0tXGsvJ70olp25VXxhRWTCOjlWPFRoSydNFYTgVJqeNiZW4kILE5xokYQO7iRQ4cKa4gJDz5jXv+BCggQZiS6vsO4pLaZnPIGVkyJHdT7u05A98z2PEKDAvjc4p6nr+60bk4SR4rrBt3n4mqaCJTyYztzK0lLjCI6vP9pHzpnIR3oxeuTwhpmjx/tkkVlZiZFkVFS59KO1m3HrP6B5ZMHlwimxEUQEx7MxowyXtlbyGcWjO93LePOTunhUivQRKCUn2rvcLAnr4qlTvQPACSNDiM0KMDpGkGHw/DAa4c4VFTLyqk9t5cPVFpSFJUNrZTVD+4J555sz7Geqh7MZHhgLae5OGUMbx0sprG1gy+uSO33PRNiRjE/OZq3D2kiUEp50eETdTS0drAktf/+AbCaZlJjI5yqETS0tHPX33fx5JZc7lw1mbtXTx1quMCpkUOubB7afqyCJam9P1XtjMX2Zzh/YgxznZieAqxawf786mExaZ0mAqX8VOdCNH09Udxdalw4Of0MIS2pbebaP29lQ0YpP75yDt//9KwhXWS7cvXIodK6ZrLLGlg+yP6BTp3NSjf1MGS0N+tmW81D7wyDWoEmAqX81M7cSpLHjGJctPOduKlxEeRXNvY6HfThE7Vc+chmcssbePzmpT2OpR+KsREhxEeFumzk0A571tXBdhR3WjxpDG985VyuXuT89BRT4iNJS4zirV76Cdo7HPz5w2ya2zqGFJszNBEo5YeMMezMrXS6f6DTlLgI2joMRdVnDiEtrG7imj9txRj4591ns3ZmgqvCPc1MF041sf1YJREhgcwZ4tBWsFZeG2iH+MVzktiZW0lZD7O6/t9/DvPTt46wMaN0yLH1RxOBUn7oSHEd5fWtA04EJ2ch7aF56Lntx2lsbef5u1YMenI5Z6QlRpFZUkfHAB9s68n2nAoWp44lqJ9FaNzlkjlJGAPr00tO2/7cjuM8sTmX28+dzLo549wehyYCpfzQY5uOMSo4kHUDnFunt1lI2zoc/HNXPmvSEk4OM3WXtKQoWtod5A1xkZqK+hYyS+pZPoA+ElebmRTFpNjw00YPbTtWwQ9eOcjqGfHcf8lMj8ShiUApP5NX0cCr+wr5wooUxkb0Pd69u/ioUCJCAs8YOfT+4VJK61q4YVmKK0Ptkas6jE/1D3gvEYgI6+YksSWrnJrGNvIrG7nnmd1Mig3n9zcs9FhNRROBUn7m0Y3ZBAUGcOeqKQN+r4gwqYchpM/tOE7S6DDWpPW94LsrTE+MJEDgsY+O8eaBEzS1Dq4zdXtOJWHBAcyd0Pusq56wbnYS7Q7Dq/sLuf2pnTgM/PXmpYNa23mwNBEoNYx8fLSci379IbXN7lnJqqi6iRf3FPD5JRNJGB02qGNMjos4bRbS/MpGNh0t4/NLJ3rkDjYsOJDvXXoWBVVNfPnZPSz5yXq+/vxe3j9cMqDlILcdq2DJpLGEBHn3Mjg/OYZx0WE88Nohsssa+OONi042wXmKJgKlhpF304vJLKnn/cMl/e88CI9tOoYx8KXVA68NdEqNC6egqok2ewjpCzvzEeDapRNdFGX/7lg1hW33n8+zdyznigXj2ZBRxu1P7eJLT+9y6v3Vja1klNR5tX+gU0CAcPHsJBwGHrh8Fuf0MmupW2Pw+BmVUr3aX2CtdOWOOWhK65p5bsdxrl40geQxzq/C1V1qbAQdDkN+ZSNtHQ5esDuJhzqp3EAFBggrp8Xx06vnsfP7F3Dz2ZPYmFlGhRPTT+zIqcQYhvwgmavcd9EMnrptGV88O9Ur59dEoNQw0dru4HBRLUEBwoeZZWesgztUj3+UQ1uHg3vWTBvScabE2yOHKhp4/3ApZR7qJO5LSFAAVy9Kxhj4MLOs3/23HaskNMhafH44iAoLZvUM9/ev9EYTgVKDVNfc5tLpATKK62jtcPD5pRNpbnPwYUbvFzRjDHc/vZt/bM9z6thVDa08sy2Py+ePH3L788lnCcobPdpJ3J+5E6KJiwxhQx+fW6ftORUsTIkhNKjvVdn8hSYC5Rc2ZZax6qEPKK0b2KIqffn1+qN86endpNsLrwzV/oJqwGr/HhsR0uvUAwBbsyt4+1Axv3o306kpCJ7YkktDawdfXju02gBY0zxEhQXx8dEyj3YS9ycgQFg9I4FNmWW9ToEBUNPURvqJ2iFPK+FLvP/bU8oDXtiVT35lE89sde4Ouj+1zW28sPM4ABtcNAXAgYJqxoQHkxobzoVnJfLBkVJa2nu+yD+xJZeQoAAqGlp5dV9hn8eta27jyc05XDw7kRmJUUOOU0SYHBfBhowyj3cS92ftzHhqmtrYl1/d6z7r00swhl6XkvRHmgiUz2tp72DjEeti/fS2PJdM4vX8juM0tHYQFxnKhiOuSgQ1zE2OsR4ymptEfUs7W7Iqztgvv7KR9w6XcNeqKcwaN5q/fpTT50Itf/0oh9rmdu5dO90lccKp5iFvdBL3ZdX0eAIDhA/6+J38c2c+qbHhJ1cWU5oIlB/Ykl1BQ2sHd6+eSlVjGy/t6fsOuj9tHQ6e3JzLiiljuWHZRPYcr6K6sXVIx2xsbSezpI759lz2K6fGEhUaxFsHT5yx79+35hIowhdWTOKOVZM5WlrfawdpYXUTf96UzWXzxjk9T74zOqeR8HYncXfRo4JZPGlMr/0Ex8rq2ZFbyTVLJrpkxTRfoYlA+bx3DxUTGRrENy6cztwJ0fz142M4hjBh2VsHiymqaebOVVNYMzMBh4FNR8uHFOOholocBuYlW0+5hgYFcv5ZCaxPLzmtvbuhpZ3nd+azbk4SSdFhXDZvPImjQ3n845wej/vT/xwG4P5LzxpSfN1duWA8d6+eOiw6ibtbm5bA4RO1FNec2R/0r90FBAj9rinsbzQRKJ/W4TCsTy9hTVo8oUGB3LFqMsfKGtiYObjmHGMMf/3oGFPiIliblsD85BjGRoScbHoarP12m/b8Lnft6+YkUdXYdnJOHICX9xZS19zOreekAtawyZtXpvLR0XIOnzi903r7sQreOHCCu1dPdXnzzZT4SL57ycxh0Unc3dqZVnLqPn1ze4eDF3cXsDYtgcRBPlXtq4bfb1EpF9qXX0V5fSsX2atBXTp3HEmjw/jrRz3fQfdnV14VBwpquO3cyQQECIEBwuoZ8WzMLBvStMgHCmoYFx122rQPq2ckEBYccHJmSmMMT27JZe6EaBalnGrfvmFZCqOCA0+rFXQ4DP/7ejrjo8P40qdcs0zkSJGWGMX46LAz+gk2ZpRRWtcyrDq3hwtNBMqnvXuohOBAOdmEERwYwC3npLIlu4JDRTUDPt5fPzpGTHgwn110qmlhTVo8lQ2tHCjofaRKfw4UVDOvWxv+qJBA1sxI4O2DxTgchs1ZFWSV1nPLytTT2rdjwkO4Zkkyr+4rpLTWag55YWc+6Sdq+d6nz2JUiH+NlRcR1sxMYHNW+Wmjrl7YlU9cZCjnuWnBnJFME4HyWcYY3jlUzNlT406byfH6ZSmEhwTy+ABrBbnlDbybXsIXlk867eL6qenxBAhOPcjUk5rGNnIrGk/2D3R1ydwkSuta2JtfzZNbcoiLDOGy+WcuVHLbOZNpdxie3pZHTVMbv3w3g2WTx/Lpue5f1GQ4WpuWQENrB7tyqwBreo0PjpTy2UUTCB6GzVne5rZPRET+JiKlInKwy7YHRKRQRPbZX5e66/xKHS2tJ7eikYtnJ562PXpUMNcumchr+4t67FDszRObcwgKEG46+/R1eMdEhLAwZcyglxQ8UNjZP3BmIlg7M4HgQOGxTdm8f6SUG5al9Pg0bGpcBBeelcgz2/L4+dtHqGps5UeXz/LbkTErp8YSEhhwcmjvS3sK6XAYrlmizUI9cWdqfBJY18P2XxtjFthf/3Hj+ZWfe9duW7/wrMQzXrvtnMk4jOHvW3OdOlZNYxv/3FXAFfMn9Dh983kzEzhQUDOoJ5cP2BPN9TS8c3RYMOdOi+OdQyUEinBjH4vB37FqClWNbTy7/TjXLU1x63KRw11EaBDLp4xlQ0Ypxhj+uTOfJZPGMC0h0tuhDUtB7jqwMWaTiKS66/hK9efd9BIWpsT0eOFOiQ3n4tlJ/GP7ceYlx1DV2EpFfQvl9a1UNLTS2u2J3pLaFpraOrhj1eQez7UmLZ5fvJPBhxllA77r3J9fzeS4CKJH9bwQybo5SWzIKOPSueP6HO2yNHUM85OjOVbewDcvmjGgGHzR2rQEHnwjnZf2FHKsvIG71/hXp/lA9JsIRCQeuBNI7bq/Mea2QZ7zXhG5CdgF3GeMqRrkcZTqVVF1EwcKavjOut7XfL1j1RTeOljM3c/sPrktKiyI2IgQwoLPbH656exJnDVudI/HmjVuNImjQ9k4iERwoKCG5X0sl7huzjjeOljMvef1PU+QiPDHLyymvrmd2MjQAcXgi9bOtBLBA68fIiIk0G/7S5zhTI3gVeAj4D1gqM/mPwr8GDD2vw8DPSYUEbkLuAsgJWV4Pb2ohr/16dbCLt37B7paPGkMb3zlXABiI0MYGxEy6NkoRYS1aQm8+ckJ2jocTndIltY2U1zb3GNHcafoUcE8eesyp443nKZ78LbJcRFMjrOW1bxu6UQiQt3WADLiOfPJhBtjvuOKkxljTi67JCJ/Ad7oY9/HgMcAlixZMvgB2sovvZtezLSESKbE990mPGeC69rR16Ql8PzOfHbnVTk9s2XnQjTzXTj9gzplTVo8OeUN+uxAP5y5bXnDVaN7RKRr3ewq4GBv+yo1WNWNrWw7VslFs3qvDbjDOdNiCQ6UAc1GeqCgmsAA8euOXXe6e/VUfv7ZuSyc6N0F6oc7ZxLB17CSQZOXeDIEAAAbXklEQVSI1IpInYj0OwG7iDwHbAXSRKRARG4HHhKRT0TkALAW+MaQoleqBx8cKaXDYU4+TewpUWHBLE0dy8Yjzj9PsL+ghukJkX730JenJI4O4/NLU/x2GK2z+mwaEuvTm22MOT7QAxtjru9h8+MDPY5SA1HT2MYfN2YzPjqMeS5s9nHWeTMT+Mmbhymsbuq3vd4Yw4GCai6e5dmEpVR3fdYIjDXJ+ZseikWpIWlu6+DOv+/ieEUjD1+7gIAAz98Fnm8/s/DIhqx+9z1e2Uh1Yxvzhsm6ucp/OdM0tEdElro9EqWGoMNh+MYL+9iRW8mvPj+fs6d6ZxnCyXERfGn1FJ7dfpy3e1hLoKtTHcXafq28y5lEsBzYKiLZInKgSxu/UsOCMYYHXz/EWweL+cFls7hs3nivxnPfhWnMT47mOy9+QlF1U6/77cipICQogLSkoS8fqdRQOJMILgamAucBlwOX2f8qNSw8+mE2T23N465PTeH2c3t+8teTQoIC+N31C2nvcPD15/edMT21MYZfrc/kmW3HuXh2kk6CprzOmb9A08uXUl730p4CHno7gyvmj+e7fTxF7GmTYiP4yVVz2JFbyR8+ONVf0NLewTde2Mfv3j/KNYuTefia+V6MUimLMw+UvYl14RcgDJgMZACz3RiXUk75f/85wuJJY/jFNfO80jncl6sWJrMps5zfvp/JymmxTE+I5K6nd7Mjp5JvXZzGf62ZqsMa1bDQbyIwxszt+rOILAL+y20RKeWkqoZWyutbuHv1lEFPDeFuD35mNnuOV/G15/YSFhxIQVUTv71uAZ9ZMMHboSl10oAbJ40xe7A6kJXyqmPl9QBM7WcaCW+KCgvmd9ctpLSuharGVv5x53JNAmrYcWb20f/u8mMAsAgocltESjkpq3T4JwKA+RNj+Pc9K0mICmW8TgqnhiFn+gi6jm1rx+ozeNE94SjlvOyyBkKCApgwZvhfXBfoXDdqGHMmEaQbY/7VdYOIXAP8q5f9lfKI7NJ6psRFEDjMOomVGmmc6SO438ltSnlUdlk9U3XpQaWGrNcagYhcAlwKTBCR33V5aTRWE5FSXtPS3sHxykau0I5XpYasr6ahIqzlJK8AdnfZXodOH628LK+iEYeBqfER3g5FqRGv10RgjNkP7BeRZ+39UowxGR6LTKk+ZI+QEUNKjQTO9BGsA/YBbwOIyAIRec2tUSnVj86ho1O0RqDUkDmTCB4AlgHVAMaYfVjTTCjlNdll9UyIGUV4iC5IrtRQOZMI2owxNd226aRzyquyyxp0xJBSLuJMIjgkIjcAgSIyXUR+D2xxc1zKTzy9NZf/ezN9QO8xxlhDR7VZSCmXcCYRfAVrptEW4FmgBvi6O4NS/uGfO/P5wauH+MtHOWzOKnf6fcW1zTS2dmhHsVIu0mciEJFA4EFjzPeNMUvtr/8xxjR7KD7lozYcKeX+lz9h1fQ4xkeH8dA7GVhLZPcvu7QB0BFDSrlKf4vXdwDneigW5Sf25VfzX//Yw1njonj0C4v52gXT2Z9fzTuHSpx6f3aZPXQ0QZuGlHIFZ5qG9orIayLyRRG5uvPL7ZEpn5Rb3sBtT+4kLiqEv92ylMjQID67KJkp8RE8/G7GGcs69iS7rJ6osCDiI0M9ELFSvs+ZRBAGVHBqzeLOdYuVGpCyuhZu+tsOAJ66dRkJUWEABAUGcN+FaRwtreeVvYX9HiertJ6p8ZG6updSLuLMCmW3eiIQ5fu+/sJeSuuaee7OFUzp1r5/yZwk5k6I5tfvZXL5/PGEBPV+j5JdVs+q6fHuDlcpvzHgFcqUGoyW9g62H6vklpWTWZgy5ozXAwKEb12cRkFVE8/tON7rceqa2yipbdGOYqVcSBOB8oic8gbaHYazxkX1us+q6XGsmDKW33+QRWNrzxPcHivrHDGkHcVKuYomAuURGcV1AMxI7D0RiAjfXjeT8voWntic2+M+p0YMaY1AKVcZVCIQkUWuDkT5tqMl9QQGSL+TxC1KGcMFZyXypw+zqW5sPeP17LJ6ggKElLHh7gpVKb8z2BrBPS6NQvm8jJI6JsdFEBoU2O++37o4jfqWdv704bEzXssubWBSbDjBgVqZVcpVBvW/yRhzp6sDUb4ts6SOGYnONeekJUVx5YIJPLklh9La0x9izyqr145ipVys30QgIot6+JoqIjr/r3JKU6u1rGRf/QPdfeOCGbR3GH73wdGT29o6HORVNDBN+weUcilnagR/BLYBjwF/AbYC/wIyROQiN8amfERWaT3GQNoAEkFKbDjXL0vh+R355FVYI4XyKxtp6zBaI1DKxZxJBEXAQmPMEmPMYmAhcAy4EHjIncEp35BRYo0Ymj6ARADwlfOmERQo/Hp9JmCtQQA6YkgpV3MmEcwwxhzq/MEYkw7MNMac2ZOnVA+OltQREhhAauzARvokjA7jlpWTeXV/EUeKa08OHdXlKZVyLWcXpnlURFbbX38E0kUkFGhzc3zKB2SU1DE1IZKgQYz0uWf1VCJDg/jlOxlkl9aTEBXK6LBgN0SplP9y5n/mLUAW1mI0X8dqFroFKwmsdVdgyndkFteR5uSIoe6iw4O5e/VU3jtcyvtHSrV/QCk3cCYRXAL8wRhzlf31S2NMozHGYYyp7+1NIvI3ESkVkYNdto0VkfUictT+98xJZ5RPqWtuo6imecD9A13dek4qcZGhVDa06hoESrmBM4ngciBTRJ4WkcsGMGz0SWBdt23fBd43xkwH3rd/Vj4ss8S6VxjIiKHuwkOC+Mp50wBdlUwpd3BqGmoRCcaqGVwPPCIi640xd/Tzvk0iktpt82eANfb3TwEbge8MLGQ1kmTaI4bSkgafCACuX5ZCXXMbl80b74qwlFJdOHV3b4xpE5G3AAOMAq4E+kwEvUg0xpywvy8GEnvbUUTuAu4CSElJGcSp1HCQWVLHqOBAJsSMGtJxQoICuPe86S6KSinVlTNPFl8iIk8CR4HPAn8FkoZ6YmOtVN7ruoTGmMfsZxeWxMfrIiQjVefUEgEBupqYUsOVMzWCm4AXgC8ZY1qGeL4SERlnjDkhIuOA0iEeTw1zGcX1rE3TRK7UcNZvjcAYc70x5pXOJCAi54rII4M832vAzfb3NwOvDvI4agSobGilvL5lyP0DSin3cqqPQEQWAjcA1wA5wEtOvOc5rI7hOBEpAH4E/Az4p4jcDuQB1w4ubDUSZA5yagmllGf1mghEZAbWKKHrgXKs5iExxjj1EJkx5vpeXjp/oEGqkenkiCFNBEoNa33VCI4AHwGXGWOyAETkGx6JSvmEzJI6RocFkTg61NuhKKX60FcfwdXACWCDiPxFRM4HdOiHclpmcT0zEqMQ0T8bpYazXhOB3UF8HTAT2IA1z1CCPQGdrkOg+mSMIaOkjhnaUazUsOfMqKEGY8yzxpjLgWRgL/o0sOpHWV0LNU1t2j+g1AgwoHmBjTFV9oNe2uGr+nRqMRqdG0ip4W5Qi9cr1Z+MYh0xpNRIoYlADUlzWwdvHCiitvn0NYqOltQTFxlCbKSOGFJquHN2SmmlevTMtjx+8uZhwkMC+dziZG5emcrU+Eiro1hrA0qNCJoI1JB8nFXOhJhRrJgSy/M78vn71jxWz4gno7iOzy+d6O3wlFJO0KYhNWit7Q525FRy3swEHr52PlvuP4//vnAG6SdqaWrrYNb40d4OUSnlBK0RqEHbl19NY2sH50yLAyAuMpSvnj+du1dPZc/xKhZP0pVIlRoJNBGoQducVU6AwNlTYk/bHhIUwIpu25RSw5c2DalB25xVztwJ0USHB3s7FKXUEGgiUINS39LOvvxqVtrNQkqpkUsTgRqUHTkVtDsM52oiUGrE00SgBmVzVgUhQQHaIayUD9BEoAZlc1Y5S1PHEBYc6O1QlFJDpIlADVhZXQtHiutYOVWbhZTyBZoI1IBtyS4H0P4BpXyEJgI1YFuyKhgdFsScCdHeDkUp5QKaCNSAGGP4OKucs6fGEhigS1Aq5Qs0EagBOV7ZSGF108lpJZRSI58mAjUgH2dZ/QOaCJTyHZoI1IBsyaogaXQYU+IivB2KUspFNBEopzkchi3Z5ZwzLQ4R7R9QyldoIlBOSz9RS1VjG+dO15lFlfIlmgiU0zqfH9AHyZTyLZoIlNPeOVTCzKQoEkeHeTsUpZQLaSJQTjle0cjuvCquWDDe26EopVxME4Fyyst7CxGBKxdM8HYoSikX00Sg+mWM4eW9BayYHMv4mFHeDkcp5WKaCFS/9uZXk1vRyFWLtDaglC/SRKCobmzt8/VX9hYSGhTAJXOSPBSRUsqTNBH4ufcPl7Dox+t5ZENWj6+3tjt4fX8RF85KJCpMF6lXyhdpIvBjGcV1fPW5vQSI8Nv3jpJVWnfGPh9mllHV2MbV2iyklM/SROCnKupbuP2pnUSEBvHKl88hPDSQ77z4CQ6HOW2/l/cWEBsRwqrp8V6KVCnlbl5JBCKSKyKfiMg+EdnljRj8WWu7g3v+sYeyuhYeu2kJcyZE84NPz2J3XhVPb8s7uV9NUxvvHS7l8vnjCQ7UewalfJU3/3evNcYsMMYs8WIMI15ru4PfvneUxz/Oobmto9/9jTH88NWD7Mip5KHPzWPBxBgArl40gVXT43jo7SMUVjcB8NYnJ2htd3DVQm0WUsqX6W3eCFZc08z1f9nGr9/L5MdvpLP6Fxt4elsere2OXt/z5JZcnt+Zz71rp/GZLg+HiQj/76q5GOD7L3+CMYaX9hYyJT6Cecm6JKVSvizIS+c1wLsiYoA/G2Me676DiNwF3AWQkpLi4fCGvy3Z5Xz1ub00tnbw++sXEh8VysPvZvCDVw7y5w+z+dr507lk7jhyyxs4UlxHZkkdGcV1fHS0jItmJfLfF84445gTx4bzzYvSePCNdP64MZsdOZV886IZOuW0Uj5OjDH97+Xqk4pMMMYUikgCsB74ijFmU2/7L1myxOzapV0JYDXt/HnTMR56+wiT4yL40xcWMz0x6uRrm46W8/C7GRwoqDntfSFBAUxPiGTBxBi+d+lZRIT2fA/Q4TB89tEt7MuvBuCjb69l4thw9xZKKeUWIrLbmeZ3r9QIjDGF9r+lIvIysAzoNREoS35lIw++kc769BIunZvEQ5+bT2SXC7qIsHpGPJ+aHsd7h0s5fKKW6QmRzEiKYtLYcIKc6PANDBAe+tw8Pv27j1g4cYwmAaX8gMcTgYhEAAHGmDr7+4uABz0dx0hSUtvMHz7I4vmdxxER/ufTZ3H7uZN7bbIRES6clciFsxIHdb4ZiVH8/bblJI4OHUrYSqkRwhs1gkTgZfsiFgQ8a4x52wtxDHsV9S386cNs/r41jw6H4fNLJ3LvedMYF+3+id/OnqqrkCnlLzyeCIwxx4D5nj7vSLM/v5ob/rKNprYOrlqYzNfOn05KrDbTKKVcz1ujhlQ/Ht2YTUhQAK/eey7TEiK9HY5SyofpcwTD0ImaJtYfLuHzS1M0CSil3E4TwTD03I58HMZw43J9fkIp5X6aCIaZtg4Hz+04ztq0BB26qZTyCE0Ew8w7h4opq2vhiysmeTsUpZSf0EQwzDy9NY+JY0fxqRk67bNSyjM0EQwjmSV1bM+p5MblkwgM0Pl9lFKeoYlgGHlmWx4hQQFcu2Sit0NRSvkRfY7AQ6obW3nrYDHvHCpmanwk9100g/CQUx9/fUs7L+0p5LK54xgbEeLFSJVS/kYTgRvVt7SzPr2Y1/efYFNmGe0Ow4SYUXyYWcYHR0p5+Nr5LEoZA8Arewupb2nnC2drJ7FSyrM0EbhJcU0zl/x2E1WNbUyIGcXtqyZz+bzxzB4/mm3HKvnmv/bzuUe38OW10/jKedN5Zlses8ePZqG9YphSSnmKJgI3+ePGLOqa23n2juWsmBJLQJfO37OnxvL211fxv6+n8/sPsnh9fxG5FY387Oq5ugiMUsrjtLPYDYqqm3h+Rz7XLElm5bS405JAp6iwYH55zXz+/MXF1DW3ExMezBULxnshWqWUv9MagRv8cWMWBsOX107rd9+LZyexYnIsdS1tp3UeK6WUp+iVx8UKqhp5YWc+1y6ZSPIY56aIiA4PJjo82M2RKaVUz7RpyMUe2ZCNIE7VBpRSajjQROBC+ZWN/GtXPp9fOpHxMe5fRUwppVxBE4ELPbIhiwAR/mvtVG+HopRSTtNE4CL5lY38e3cBNyxP8ciawkop5SqaCFzk9x8cJTBAuGeN1gaUUiOLTyeC0tpm8isb3X6e7LJ6XtxTyA3LU0gcHeb28ymllCv5dCL42dtHuOBXH/K794/S3NbhlnNU1Ldwx1O7iAoL0tqAUmpE8ulE8K2L07hgViK/Wp/Jxb/ZxIaMUpcev7G1ndue2kVRdROP37yEhCitDSilRh6fTgTjokfxyA2LePr2ZQSKcOsTO/nS07sorG4a8rHbOxzc++xePimo5vfXL2TxpLEuiFgppTzPpxNBp1XT43nr66v49ro0NmWWc/7DG3lkQxat7Y5e39Pc1sHzO47zws7jVDe2nvaaMYbvv3yQD46U8uMr53DR7CR3F0EppdxGjDHejqFfS5YsMbt27XLJsQqrm/jx6+m8faiYKfERPHjFHM6dHnfy9bYOB//aVcDvPzjKiZpmAIIDhU9Nj+fy+eO5cFYij206xm/fP8pXzpvGfReluSQupZRyNRHZbYxZ0u9+/pYIOm3MKOVHrx0ir6KRT88bx/2XzGRHTiW/ee8oxysbWZQSw30XpTE6LJjXDxTx+v4iTtQ0ExoUQEu7g2sWJ/PQ5+bptNFKqWFLE4ETmts6+MumY/xhQxYtdjPR7PGj+eZFaaxJiz/tIu9wGHYfr+L1/UU4jOFHl88mONAvWtaUUiOUJoIByK9s5OlteSyYGMO62Uk9rh+glFIjjbOJQKehBiaODed7l57l7TCUUsortG1DKaX8nCYCpZTyc5oIlFLKz2kiUEopP6eJQCml/JwmAqWU8nOaCJRSys9pIlBKKT83Ip4sFpEyIG+Qb48Dyl0YznDga2XytfKA75XJ18oDvlemnsozyRgT398bR0QiGAoR2eXMI9Yjia+VydfKA75XJl8rD/hemYZSHm0aUkopP6eJQCml/Jw/JILHvB2AG/hamXytPOB7ZfK18oDvlWnQ5fH5PgKllFJ984cagVJKqT5oIlBKKT/n04lARNaJSIaIZInId70dz0CJyN9EpFREDnbZNlZE1ovIUfvfMd6McaBEZKKIbBCRdBE5JCJfs7ePyHKJSJiI7BCR/XZ5/tfePllEttt/ey+ISIi3Yx0IEQkUkb0i8ob980gvT66IfCIi+0Rkl71tRP7NAYhIjIj8W0SOiMhhETl7KOXx2UQgIoHAI8AlwCzgehGZ5d2oBuxJYF23bd8F3jfGTAfet38eSdqB+4wxs4AVwJft38tILVcLcJ4xZj6wAFgnIiuAnwO/NsZMA6qA270Y42B8DTjc5eeRXh6AtcaYBV3G2o/UvzmA3wJvG2NmAvOxfleDL48xxie/gLOBd7r8fD9wv7fjGkQ5UoGDXX7OAMbZ348DMrwd4xDL9ypwoS+UCwgH9gDLsZ7wDLK3n/a3ONy/gGT7QnIe8AYgI7k8dsy5QFy3bSPybw6IBnKwB/u4ojw+WyMAJgD5XX4usLeNdInGmBP298VAojeDGQoRSQUWAtsZweWym1H2AaXAeiAbqDbGtNu7jLS/vd8A3wYc9s+xjOzyABjgXRHZLSJ32dtG6t/cZKAMeMJuvvuriEQwhPL4ciLwecZK/SNy/K+IRAIvAl83xtR2fW2klcsY02GMWYB1J70MmOnlkAZNRC4DSo0xu70di4uda4xZhNVU/GUR+VTXF0fY31wQsAh41BizEGigWzPQQMvjy4mgEJjY5edke9tIVyIi4wDsf0u9HM+AiUgwVhL4hzHmJXvziC+XMaYa2IDVdBIjIkH2SyPpb+8c4AoRyQWex2oe+i0jtzwAGGMK7X9LgZexEvZI/ZsrAAqMMdvtn/+NlRgGXR5fTgQ7gen2aIcQ4DrgNS/H5AqvATfb39+M1cY+YoiIAI8Dh40xv+ry0ogsl4jEi0iM/f0orP6Ow1gJ4XP2biOmPMaY+40xycaYVKz/Mx8YY25khJYHQEQiRCSq83vgIuAgI/RvzhhTDOSLSJq96XwgnaGUx9sdH27uVLkUyMRqs/2+t+MZRPzPASeANqy7gNux2mvfB44C7wFjvR3nAMt0LlaV9QCwz/66dKSWC5gH7LXLcxD4ob19CrADyAL+BYR6O9ZBlG0N8MZIL48d+37761DntWCk/s3ZsS8Adtl/d68AY4ZSHp1iQiml/JwvNw0ppZRygiYCpZTyc5oIlFLKz2kiUEopP6eJQCml/JwmAjViiUisPZvkPhEpFpHCLj87NTumiDzRZTx2b/t8WURudE3UPR7/ahEZsU8jq5FPh48qnyAiDwD1xphfdtsuWH/njh7fOAyIyDPAv40xr3g7FuWftEagfI6ITLPXO/gH1gNE40TkMRHZZa8Z8MMu+34sIgtEJEhEqkXkZ/baAltFJMHe5yci8vUu+//MXoMgQ0RW2tsjRORF+7z/ts+1oIfYfmHvc0BEfi4iq7AeqPu1XZNJFZHpIvKOPUHaJhGZYb/3GRF51N6eKSKXuP/TVP4gqP9dlBqRZgI3GWM6FyH5rjGm0p4vZ4OI/NsYk97tPdHAh8aY74rIr4DbgJ/1cGwxxiwTkSuAH2KtGfEVoNgY81kRmY81HfXpbxJJxLrozzbGGBGJMcZUi8h/6FIjEJENwB3GmGwROQf4A9a0CGDNn7UUmA68JyLTjDEtg/+YlNIagfJd2Z1JwHa9iOzBukCfhbVYUXdNxpi37O93Y60F0ZOXetjnXKxJ2jDGdE5l0F0l1tTOfxGRq7BmjTyNPW/RCuBFe2rrR4DxXXb5pzHGYYzJwJpmfXovMSrlNK0RKF918iIrItOxVtxaZt+BPwOE9fCe1i7fd9D7/48WJ/Y5gzGmTUSWYE1Mdw1wD6fu9E+GC5Qba1rrHg/Tz89KDZjWCJQ/GA3UAbX29LwXu+Ecm4FrAURkLj3UOOwZMEcbY94AvoG1KA92bFEAxpgq4IRdY0BEAuympk7XiGUGVjPRUTeURfkZrREof7AHa5reI0Ae1kXb1X4P/F1E0u1zpQM13faJBl4SkVCsm7D/trc/B/xZRO4DrsSa/vlReyRUCPAM1syZYK0DsAuIBO4yxnStxSg1KDp8VCkXsDuhg4wxzXZT1LvAdHNqeUdXnEOHmSq30BqBUq4RCbxvJwQBvuTKJKCUO2mNQCml/Jx2FiullJ/TRKCUUn5OE4FSSvk5TQRKKeXnNBEopZSf+/9A4YI84QIwMAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "eval_score = [l['average_train_unwr_rews'] for l in all_logs]\n",
    "plt.plot(eval_score)\n",
    "plt.title(f'Learning curve for {env_name}')\n",
    "plt.xlabel('Training step')\n",
    "plt.ylabel('Avg. return')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d56bd690",
   "metadata": {},
   "source": [
    "# Plot estimated vs actual Q-values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0117e017",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('avg_return', 4.4)\n",
      "('std_return', 5.1613951602255765)\n",
      "('max_return', 19.0)\n",
      "('min_return', 0.0)\n",
      "('mean_pathlen', 57.6)\n",
      "('n_paths', 10)\n",
      "('n_transitions', 576)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsvXmcbOtZ1/t91lRTV8+99z7n7DNkJINAiMeEyHREBIUQLjIIyHi5N8J1wCCCoBBv5EYuoKKiMgQFgSQyGxGFXDEBBEzOCZB4A5jpTPtMe++ealzT+/jHu1Z1de8eqrprVVV3r+/n05+uYdVa73rXOz7v730eUVVKSkpKSkqOw5l1AkpKSkpK5p+ysygpKSkpOZGysygpKSkpOZGysygpKSkpOZGysygpKSkpOZGysygpKSkpOZGysygpKSkpOZGysyi5kIjIj4vId886HdNARD5NRP74mO/vE5G2iLjTTNdpEBEVkRfOOh0ld1J2FpcAEXlURHoi0hKRbRH5bRH5BhEZ6fmLyANZJfbOkAYRkb8tIh/K0vK4iLxZRILTnvOiIiKvEpFfyZ7Vpoi8R0S+7qjjVfU3VfXjhn7/qIh81tD3j6vqgqqmp0jLdRH5aRG5LSKdLC2fO/5dlZx3ys7i8vD5qtoE7ge+B/g24MemeP1/Brwe+GqgCfwF4LOAt08xDXOPiLwG+HXg3cALgTXgG7H5ddjxp+7AR0jLKvBbQAS8HFgH/gnwdhH534q6bsmcoqrl3wX/Ax4FPuvAZ68CDPAnsvefB/wesAs8Afz9oWMfBxRoZ3+vAV6AbdRuA7eAnwaWj7j+i4AUeNWBz+8FQuAzjvjdHwKvHXrvATeBV2bvfxZ4BtgBfgN4+dCxPw58d/b6a4HfOnBuBV6Yva4A35/d57PADwG17Lt14JeBbWAT+E3AOSSt/wr4/gOf/Xvgm7PX3wbcAFrAHwN/9oh7/i3gXxzzLB8CnszO9wzwk/ln2fc/mT3XXvasvhV4ILtfLztmFfg3wFPAFvBLR1zrHwD/4+D9Ztf+KCCH/ObVWbrcoc++EHj/ULn7nSw/nwZ+EAiOeC7vAv6Poe/2PUfgJcA7s+fyx8CXDn33ucAHs/y+AXzLrOvhef8rZxaXFFV9D7bR+bTsow521L+M7Ti+cWj0+OnZ/2W15ozfAQT4h8DdwEuxDf/fP+JyfxbbmL3nQBqeAH4X+Owjfvc24MuH3n8OcEtV35e9/0/YjugK8D5sh3Uavgd4MfAK7Gj+HuC7su/+FjafNoCrwHdgG7TD0vqXREQARGQFe19vF5GPA/4a8KfUzu4+B9uB70NE6tiO+OdOSO81bIN/P3a2NkBVvwrb6X1+9qy+95Df/yRQx84WrmBnC4fx54CfV1Vz4POfAZ6Hzat9qOp/x5alzxz6+CuAt2avU+AN2E74Ndiy8X8dcf0jEZEGtqN4a3YPXwb8SxF5WXbIjwF/JcvvP4Ed2JScgbKzuNw8hW10UNV3qeoHVNWo6vuxjd9nHPVDVf2wqr5TVUNVvQn842OOX8eOIg/jaWxDfBhvBV6XNaJgG523DaXhX6tqS1VDbEf1iSKydFSaDyNr3F8PvEFVN1W1BbwZ2/gAxMBdwP2qGqtdHziss/hNbCeSd75fDPyOqj6FbSArwMtExFfVR1X1I4ecYwVbJ4/KqxwDvDHL+97odwsichfWpPUNqrqV3dO7jzj8qOeWf3bUcxt08iLSxI7y3wagqo+o6u+qaqKqjwI/zDHl7BheCzyqqv8mO9fvAT8PfEn2fYzN78XsPt935JlKRqLsLC4392Cn8IjIq0Xkv4rITRHZAb4B21gciohcFZG3i8gNEdkFfuqY429hG9zDuAu4NaTYaYtIG2yHhDVFfX7WYbyObIQqIq6IfI+IfCS7/qPZ+Y5M8xFsYEfZj2QLytvAf2avIfw+4MPAr4nIR0Xk7xx2kqwDeTt7M6GvIJvpZPfxN7Ed2nNZvt19yGm2sB3BUXmVc1NV+yPe30HuBTZVdWuEY496bvlnt0TkO4ae2w9ln78V+IsiUgH+IvA+VX0MQEReLCK/LCLPZM/tzYz/zMDOql6dP7Psuf1l7KwL4IuwndRjIvLubC2o5AyUncUlRUT+FLaz+K3so7cC7wDuVdUlrN1esu8OG0m/Ofv841V1EfjKoeMP8uvAvSLyqgNpuBf4ZOBduqfYWVDVhaHD8lHqFwAfzBpesI3xF2AXyZewdnmOSEMH2yHk17029N0trH3/5aq6nP0t5WnIZi5/S1Wfj+2svllE/uwR9/k24ItF5H6s7f7n8y9U9a2q+qnYRk6B//fgj1W1i7Xnf9ER5x8ceobvnwBWRWT5hHMA/H/YRv9gO/GlWNPch1X1zUPP7RsAVPWDwGPYGcywCQrs2s4fAS/Kys13cHS52ffc2OsI8vt499Azy02k35il4b2q+gVYE9UvYU1nJWeg7CwuGSKyKCKvxY6Cf0pVP5B91cSOOPtZo/4VQz+7iR3xPn/osyZ2AXVHRO4B/vZR11TV/4ntfH5aRD45mxW8HNuY/ja2UTqKt2Nt/9/I/kaniV0cv41tUN58zDn+AHi5iLxCRKoMra1k9vgfBf6JiFwBEJF7RORzstevFZEXZuaqHaxJ6aANPz/X72E7n7cAv6qq29k5Pk5EPjMbafexndOh58AuSH9tJjNey37/iSIyjmrsWfY/q+E0Po1d6/mXIrIiIr6IfPphx2LXMpaAHxORayJSFZEvB74TawY76h7APqtvwq53/ezQ502siKItIi/BPtej+H1sZ1UXu/fi64e++2XgxSLyVdk9+CLyp0TkpSISiMhfFpElVY2z6x2X1pJRmOXqevk3nT+siaaHVYbsYEevf5X9ipUvxo4GW9iK+IPYziT//k3YTmMbOxt4OfAItsP4fbKF4GPS4GBVNB/GNvKKbUSWRkj/fwES4NrQZwtYtVErS/dXs19J8+Nkaqjs/d/FNuRPYGdBw8dWsZ3NR7ENyx8CfyP77g1Z/nWwo+nvPCGt35md+0uGPvsE4D1ZWjez/L37mHO8Ctug72TH/3fgq7PvHjqYzwc/w864Hs+e1bdwuBrqJ7CdyhbwC8ek5T7sjGkzewYx8DUjPLP7sA30fzzw+adjZxZt7DrPm9ivcBp+LuvAr2X59t+wnfzwsR8H/MesXN7GzmBfAQRYU+JW9jzfC3zqrOvhef+TLNNLSqaKiPzfWEnlp2s2Ai+Zb0RkEdto/6KqftdJx5dcLEozVMlMUNU3Aj+CnaWUnANUdRe7aJweWPcpuQSUM4uSkpKSkhMpZxYlJSUlJSdSmF+ZabO+vq4PPPDArJNRUlJScq545JFHbqnqURssB1yYzuKBBx7g4YcfnnUySkpKSs4VIvLYKMeVZqiSkpKSkhMpO4uSkpKSkhMpO4uSkpKSkhMpO4uSkpKSkhMpO4uSkpKSkhMpO4uSkpKSkhO5MNLZ06IKRkHE+kke5zVqvZ6d9Bqx17Eh1I5/7Yj9yWlfk6XvLK9PkxdlvoySL0onTHEdoeo7KFLmy5TKS5ikRImyUHUBuRD5kt3uvnuW7PpFcKk7C6MQpbNORcllIDXK4zc7g0p+/0Yd1ymwZpcMeHa7T7ufIAJrCwFLdb/YVnWGVNzibu1SdxalW6ySaaEoa80KnX6C6wrOBW2s5pFG1cVxhChJaVS9fJJSMiaXurNwBFyBtOw0SgrGE6FZ82jWbJWTsrOYGo2KR6Oy9/6iZr3vlGaokpLzj8jERrOqum90LLnhuuRQhjtmVbVrAplB8CLlXdGWkkvdWRgtZxXjYFRRpbS1n4K8kbKLkGfLv3Y/oROmJKnh6lIVz5ULO1qeNJ0woRumhLHh2vLFyrtEwdVyzaIQLkohmQZb7YjNdgTAykLASsMvTSkj0urF3NwNUYWFmseVxcqZ8m6h6uF7Djc2ezx2q4vnCveu1XCdUgl/Eo2KR73ikRrFcyY325sHir6XS126HIHgHOTAPASo2u3FVloINCpu2VGMQaufYDK5Yz04e96JCHFiBvrJwHPK53EEB+uOiBUXeI7YPLtA+Vb0msU5aCqLwyhEZtapOBpVJUkNYWJsoZ9hp7HSCAbmp04/mYsO7LywVPPxXZt3vSidSN7VApdq4AKQJAZjyucxjKoSD9cd7uw0LhqRKbaJuNRmqHlv73pRSi9KiVNloxngzHDavFj3adas7LCUfY5Ho+rRqHoY1Ynlnec63LNaw6giXMzG7yz0opRulJKmyvpigMP5X5swRtntxXTDlIWqx0LNm2pdvNSdxbxLZ+sVj1o2epTBNtbZIRNU9FxGiqjYZcd9OPNWdybBZidiuxMDUPGdO+piKZ0tmHkvQvtHjGXDUFIyKhet7jiZOxDFmtAPYhScUg1VDEYPz/R5QFUJkxTPcfDcS720VFJSgl03rAUe3TChUb2z6U612Ab9UncW8zqD70QJv/WRm8SpUvGEh150Fb/sMEpKLjUiQi1wB+a1O74v+PqXugVyxNr55o3dfkyUGlJVGoE/F3bpWaqfSuVVScnJBAU6EYRL3lmkBuI5k86qKs3AY7ni4whEaUqUzs41rlHFGKUbJcQDGeJ0UFX6cTokNy07jZKSowjTUjp7qYgSQ5wqL726xErDt3LZGc4sUqMkqdINDRqA506vyISJoRumGKNUfXfgu/+0dMKE7U6Mk7mqDvzDp/MlJSV3cqk7i9wMNU+zi4rvEnh2wrcn+ZudU2XfFXzXoeoPp2k6VH2XyoTyIjXK01t9ey7g2vKlnlSXXEDOtRlKRB4VkQ+IyO+LyMOHfC8i8s9E5MMi8n4ReeXQd18jIh/K/r6mqDQmc9RR5IjY2YSqYrJdmbMzw8i+NE396hPKC2FPeojMrwqupOS0pBdgB/efUdVbR3z3F4AXZX+vBv4V8GoRWQXeCDyIbRUeEZF3qOrWJBOW++uZR1SV2+2IfraD+9612qWW0LZ6Ma1+QpQo963XcGW8vHAc4YGNBu0wwRFBUOLEPn3fu7z5WnJxuOjS2S8A/q3aoeLvisiyiNwFPAS8U1U3AUTkncCfB942yYvPgcjoUKLE8OTtLkbBc4T71muZIuryxvhaqPks1Hwgz4Hx88JxhMWaj6ryxO2edcYHXF+rUSnXL0rOOeddOqvAr4nIIyLy+kO+vwd4Yuj9k9lnR32+DxF5vYg8LCIP37x5c+zECfMpnY0SM5hO+plHUWsCupwdBVi3Fk7mbuSseaGa5TG2gBbdUZTS35JpcK7XLIBPVdVXYs1Nf1VEPn2SJ1fVH1HVB1X1wY2NjbF/b3S+Frdz6oHLYs3DEdvQpKWBfcAk1k0cR1hvBniu4LlCN0wmkLI7Mdmza/cTumHpqbekWM61dFZVb2T/nxORXwReBfzG0CE3gHuH3l/PPruBNUUNf/6uItM6T4hYr6KuYxuzyzuf2I+qst2J2e3FVHyH9Wbl1Os4y42A5UYw4RTuR9WqsOLUMIlxmTHKdiei1U+oBi7rCwHuJV7HKpkuhZU0EWmISDN/DXw28D8OHPYO4KszVdQnAzuq+jTwq8Bni8iKiKxkv/3VSadxXndwd8OUzXZEnCqpsSPhkvOWL4rr2MBEK42Ahao3kXCqW52YOLUbJWWu779k2hRthipyZnEV+MWsgnjAW1X1P4vINwCo6g8BvwJ8LvBhoAt8Xfbdpoj8A+C92bnelC92T5p5lM56bi5XhTQ15cwi4zzki2ZxykXyJXidmOR4cP/YGcs83n/J7EhMsW7K5aLYUR988EF9+OE7tnIcyzy6+8iJkpRndvo0qz4rBZtLzhOpUbphQuA5c6lgimIbdGerHVGveFxZOlu87YOkRumECdWhzZsl00VVBwKJypyFtK2cYnYhIo+o6oMnHTdr6exMkSH/8PPEbj/mNz58ExGo+g4PvfDqIKTpZcd1hGYmoZ1HfM9h0XVYqheTRjeT/5bMjsdvdUky0cn11drcxEAvuom43J0F4M2Zuw+AbpSgWLt81Zu/0XPJ0ViZsx19ylmdWZXMJfFQaE1/TjoKsG3ZeZbOzjXzKp3dWKjyoo0mNd/Fd5yZep0tOR2XfV/MRebacpWK5xB4Dv1ofupmdJ6lsyWnQwTW6xUqYh3pec6l7tOnSrufsNmOUFXWmwGN6mxNPqlRbrdCOmFCzXfZWKqWJskZs1D1WDgkUl0RRIlhsxXSi1OaVZ/VZjCz+DaXurNwxMrN5mhwAEAvTNnpxIMdxmXjMD2e2e4PXleD2VePVi9mt2c3DYojhdulzyf5cPriZc5WO6Id2gbqpD1X530H91yjQDxnHQVkju3EPvgknUM72QWm4jtW+CBWkTZrtWDgOZmLEwa+rEr2MKrWIehMPTMXR9Xfe/7hCTbz5AJ4nZ1bdE69zgaew/OvNOhGKYHnzEVY1YuMqtKPUhS4Z6VKlNq9ElV/9ouX9YrH86406EVp1pGVZWGY3V5MqxuTGrj3FN6IZ4mqEiYGkyqVwD3UgrDUCGhUPcLYUAvcY59/0V6BLnVnkcc3mMcOQ0RoVC7145kaj97sYrIh2X1r9bnTzjuO0JiSjfy8sVTzWaz5Q+aZ8+OZ+bmdkHbfmhjXmgFLdf/Qcue5zkhubdxSOlss8yidzU0f89RgXWSGHTW6pS+uc4VksUmydxM773AdLKo+xqkZpHwSGyzdgqWzl7qzmEfpbJSk9KKUJFVWGoFd0Cw7jUK5e6XKVicGIIxTanOwsF0yDpOtH1FiiBNDL0ppVD2MKr0oZbHm47uTixh5ZanKVicijE22Nnm2PVVRerod3KNS1oo5w3McKh64juEySfWT1HBzN6QbpdQDlytLFdwpSYbrFY/6ISa/3W7MViciNcrGYmUizgBL5h/PEfAcUlV8V1AVUk8nrkoMPIerS9WJnrNILnVnkZsB52l24ThCNXBRHbKb557pLjDtfkInkwi6jszFov7NVjhQl5y0uHgHl+CZXVQcRwgc2TeL8CY4oyiKImcVcMmls/NohsrJbaV70sCLJwscpuK7AzcZvTglNZotOs/unuuBO5At9uPRZLSq1n14qpp5oL24z+yiM9w5zHtHAeUO7kKZ93rc6sV0wpR+bLh3rXbqQD/ngVrg8vwrDf742RaP73T58O0Wn/HCKzhTvmejmkW1szZlk3VaozqLixNDN0rZ7sTUKy4bi5UppLrkvKCqdKMUY5Ra4E60ThfdnF3qzsIR+zevUUubNZ+Fqj80tTw/ssDTICK8+GqTF+lCpnKBad/zY7mMVuHedTczRYxeoX3PYcmzXmfntFiVzJCnt/r0MpcRV5cqNKqTM2+V0tmCmcfOwqhm5o+9XSDDr6dLFsJnSnJeZ9hv/Aym/iYLXgR7a1rDpqST7n94nWkcr7MXRy599vIyV3mhuq/WndWTcGr2zjdKSNxx8sKRUjpbGEbnL1LeVjfiuXafME55/lqTODHEqWGtWSm8MBxGnChxmtLuJ9QCt3BFkHXtLTObQN2zWmOnE2OyADeuI8SJ0u4nLNQ8u44xyv2PmEdGlTA2dPoJCKwtBIc2SIN8mXPOUl6MsTua270YR4TVZjDzew4TQ5QYumHKcsPPzJGnP9+15Srb3ZgoMaTm6NjsqTFEidLuZ3mxcHJexKbYDuNSdxbzyFKm5Y4SQ73ikngOaWoGu82njecKrmsXev0CI7PFieG53ZBelFILXK4tz8a7atV3qS7v6d2N6iBOQKWAgO1C7lbEO/T5zku+jMpZykse7Au8uVDDgZW3eq6DKzKR8u97zkjrWI4IVV8AD3dO8uJSdxau9ddHNEezC0eEhYqPBnYkGXiCzlC2l68dVMeVjo5JJ0wGtlzfnR/vqrlZbGzp7Ijk5zzKxchwvnhzlC9HcZbykh9f9Ysta+MgIrgCtcp003SavCilswVidL46CtiTXhrd/zdr6WzRFaWaOVIT7NR/3pRqRd//UecfzpdoDvPlKM6SX/PSUQwzy8HaqJTS2QKZx4rX6iV0QrtBzXeFJFU8V7hndfrS2cQYnt7pEaWGq80aCwU6Nqz6Lg9s1IkSU1hM4yQ1tPsxgrBQ8+fWnKOq9GNDP/M0e/96jTgdXb5bcjk599JZEXGBh4EbqvraA9/9E+DPZG/rwBVVXc6+S4EPZN89rqqvm3Ta5lE626x5eI7QCdNBA3F9rTYTGelvfeQW7TBBVVmuBTQKNkWJCBW/mJjjxiiP3exmF4JmfbYR8I5jt5dwazdEgWbVoxZUClkvKblYeBdADfVNwB8Ciwe/UNU35K9F5K8DnzT0dU9VX1F04uZtnJYrgXL1aL6wPYsRpVHFYKV+s1pk2y8dPH1nOWzEE52/5z7MfqnuDBNSckZ0n/XCityKHGwVdmqg4DULEbkOfB7wlhEO/3LgbUWm5yBGIc0e5jy5ZqgFLtdWqjQqLvWKO7OZz2uet86L1pvcs1QjnYF0sxMm3GpFPLvTx5izRapyHeH6Wo1m1WOx5pGkxWTqJMrRUt3nylKFesWl6rtzaS4tOZkok1w/vd2n1UsKNxPF5zxS3g8A3wo0jztIRO4Hngf8+tDHVRF5GEiA71HVXzrkd68HXg9w3333jZ24vO3b7kRstiOMWp37cuPwICTTIjXKVjuiH1uN92LNZxZj4Zrv8tJrd0wIp0Y9cAk8J9tjwJmyQFXZ6cQ22IzAQs3Dn/BYqdWLudWyXmqX6z5rp9wnICI0az7N2vyaykpOxncF3/UG8tvzTmF3ICKvBZ5T1UdGOPzLgJ9T1eGI2Per6oPAVwA/ICIvOPgjVf0RVX1QVR/c2NgYO42OQODAdicejN4bc+CGuh+bQbxdz3Vw5mAhdhazLhHBdx18N1/YPdvO2VZ/b3RXLWBtZLsTDwIpNaYstbyoHCx38zL7HwURyeTvjlW0jVgeTnOPgm3Lzqt09lOA14nIo8Dbgc8UkZ864tgv44AJSlVvZP8/CryL/esZEyGXzi7W9/wvdfvJzAtk1XcGG4DixDodmxW5WSVOraR3Vp3GWXEdoV6xm8VQG+Ro0jTr3mAfRCcazUttyeHk5S41SpIaVO3/1MxeRj4u43QSqkpihu9zNJRzbIZS1W8Hvh1ARB4CvkVVv/LgcSLyEmAF+J2hz1aArqqGIrKO7Xi+d/JptP9XM9OTMToX00XPdbhvvU6SmrFGJJMmNcqt3T7dyFbW1QUbJ/g8IiKsNytsOxGuQyHPebkesFTzSeekHIF1rd7uxbiuw+KQXDhKDLu9GEdgsebPTXpzFBvjpN1PEGCh6tHOZoZ3LVeZb4nC6UiN0osSWr0U33NYbwYYVTr9hH6c2iBdxygSz7109iAi8ibgYVV9R/bRlwFv1/3d6EuBHxaR3HnK96jqByedFkesp8ZU7U5dp2i3jWMy6wp8Y7NHnNg4wfes1s61fDM1yuO3uoP3KwvFuA4XEbw5KUdRYrhxu4di7efLWUefpIYnbnUHarvlRjDTdB6GI8Jizbe+pbD52hh6fRHxHGGhaj1N59xuRex244EisR4cbT71L0IMblV9F9aUhKp+14Hv/v4hx/828PFTSFrJGBRVUTXz7JmfedLXOKhQmoem5qCJoagGUO94cb4Y9hFVpL+oO8sgTL2kDNzyn46irZ6Xegf3sHR2HjHGqoBmNZK6Z7XGTjciTDJHhgWlIzc3GKOsNu0GtEk2DN0woR/be6h4DtXAxRid2ExSM7fmowoRjFE6YUKrl+AIXF2uDs5jlInldeA53LtWY7cb43sORu1M2nMd7l2vs9uNcRwmmhd5LJB5EGWMQzdM6EWGXpRwdamKf0bvspNivRlQ9R26UUrlmFkFQKLgFhjN91J3FvNQGA4jSQ1PbfUzF9lw71p9JiapD99s8aGbbRTlZdcWef7aQiEdxkLVY6sf868ffpJ2mHB9ucY3/un7qHiTUSzVKx61CizWvEE+Tuo+ttoRm50IVVht+KyM4EpaxN7zsHPCXpTyzHafNIugdtdKdSIdZsV32Vi6Mx8Dz2F9wlH8NtsRW+0IZT4k6OOQl5E09WZu/h1mHBl10Tl9qTuLXDo7b84E+7EhzgJtBJ47s1Hao5sd0mxue7VZLaziiwgf2+zRChOMwrVmBc+ZXIXNPaGKO3nfSru9eDD9r48ou86PcZ29150wGchuq74zF6aycclt6zAfEvRxKLKMnIXhOCYnxTQpes1ifrrQGTCPXmfB7uDOF5PDOKWfyzCnJMU0mWTxarM6+OzxzS6pMYXIQVWVF67V2WgECHCrHRKlk38wByta7uG3H6VDnn3HY6XhD+Sy3XA8uexwehaqHn5mCgrj9FwuMSwP5UVnDiTop2FeOop8nS1K7MDRytf3Xh9GdF6ls+eBeS3LjsBKI+Dp7T5G7Wavimc3502jKLfDhFY/ZrsX4wnUPJc4tmqimu/SrPujR4wbgW6Y4ovwdQ9ep5lFoytiUt2PU3a6NvLYct1KXMM4pRvZKGin2ahXr3gsJYYwMWdSi1V9l/s3GoN1qlY/odNLqFc9mlVvIrPLXpSy241wXYflmo83gWA+cWLY6cWkqWGxHvC8K42x1m8mgarSCVNavZhq4O6TCJ9XjCq9yEZQFLHlLH89SvCkIrjUncWwdHaeyO3XYE0S11aqmRJpOtdfrHg0Kx5/9FyLRKEVp2wIpAbaYTrxvRaNqke9YhvqokZ2qVGevN2z18AuHHquUPEdFrP7Oc2ln97qE2Ymw+XG2dPpOMJ2J+J2y9r+XVdYrJ29msaJ4camvX/XMawtTEYue2OrN/Czld//tM2m7X7CczvhYDa2fE73Ag3jOg4LVYfGUL1onFBHLoR0dp6Zs35isGvVZNOemWzezuy3niO4IlkAJsi9vhaRpmG7rMlss3sdpAwUR3llGFfaeNBZyKRGv5mTYOvVdkL5MtwYTO6ce+nkgEz0LDgF3P/4aZh8fs0Lw2XhpIGUUSjGwb/lUncWRucrlgXA+27s8IGnWzy51eOL/sQ1JLEbqGYR/OhTnr/OR55r81yrj+86BK5DveplssJi9ls8ervDMy07Wv+0F6zjinUkuN2J6ccpUWK4vlYf22W64wj3b9Rp9exU/rjHnho7U3BHWGS/e7VGp58QJWZi+bJYs87nOmFy7CascfBch/sIZM26AAAgAElEQVQ26rR6sRUPTKi3uL5apx0m1tvAjML/Nqoe19dqtPtJIT6/Zk0+gDzJm0Oq4JXS2WKYk7WsfXziXYt8/LVFEmOsEkpsYXHOGM/hNPzRs7t87HYHVbhrqcb19XphG/P6UcpTWz1So9y1UOOuFau+ipKUG5tWUuo5NmKgnRCMnxee67Bygvnl6a0e3dD6jbq2XLX+pI65XyeTNk4SEaEWuNQm1FHk+K7D6oR3rjuOZF6RZ0vFdwsLnDVLWr2Y53ZDUKhXXK4tH61KLKWzBeKItfPFc6SIymcPvu6NIpTZqDSe2ukPZl7Xliaj+z+Kfpxi1N5nPXAJMgljGJuBI8XAdyY2ej1KhtgJ9xwM1gqODDiOLLLkeC5S/g3fSydMB6a1k8pj4M5YOisiXyIizez13xORXxCRVxaXpOmRmvnqKMCqIHb7cbbRS2daCV52dZGab2c3z7b6hUkhjSqJKv3ENtTDHm7t1Nsel6TmzB54U6OEScrNdn/gxbST7R5XVdaawSDcbicsRv6ZP9d+nNKPrdw2HHo9fytp88tFyrtBuYhSoux+lhs+lUy1FibHl/8wnb109jtV9WdF5FOBzwK+D/hXwKuLS9blZacX81yrx0dudagHHmuNgBdfaU5sN/M4XGlW8UWIUz3RfHMajFF2ejFPbHZpRzELNZ/nbdRxXbsc3Y9SelFKs+az2sjkkGfoOLthwhObXVphQoriil2H6UUpnmvjDqw0goGapjB/TWor/m43ptVPs2vBUs2n0pw/p36zxBhlpxvTCRMaFY+lur9PmBDGhk6YYFSp+O7cbGaMEsN2J7J1p+FTr5zc1Bq1v2uHCZ7jsOw5VH2Xe9frczFzGqWzyOflnwf8iKr+RxH57gLTNDUcAc+BZI5mFyv1gDBJSbXNTj/Gc2Siu5nH4emtHnEmi/QKkEPu9GJutyLqvse1xSrrzcq+hqA6tDnxrBUlNcpTW31ccVip+jz/6sLgu4Wql12DiVzrJBzHrkm0+3vXWa77rI7gKuSysdWJ2OrEAFR9vWOscGcZyWcXs83HpzZ7JNksYGMxGKmxd7NyUT2kzI9SLmZuhgJuiMgPA38J+BURqYz4u3PBvKmhVJXAdQbuo1O9s4JMi9wEJDAo+BM/f/Y634x2kDza2FnJZbi2ObHyXx18nl9juhntOc6++y+5k+Eykh6RR/nzszvy7cxt1iap4bqTjriRa9g7sjJ+xLx0DnZwfynw54HvV9VtEbkL+NvFJWl6zKN0ttVL6IcpL1ppoo6y1qjMbIx0z2qNVs/KQoMC5LKLNZ/Ac9jtxjSrxWotHEd4YKPBs9t9wjjl0Zsdri1XaYxgHiiK5YZPreLSi1LqgWujnSWpDaVbzjAAG2ujFrh0szw6rgy2ejGtfkKcKPeu1XBn6BDw+lqNTpiSGiXwRxNKJKnSDRO2ujFVzxl4Ix6VVItVLJ14blXtishzwKcCHwKS7P+5Zx7r40LNo+I79DKnVf1+ii7MJq3P7Ya0ewkI+J6VSE6yw+iGCU9nO9XjxHD3am1i5z4M1xHuWqkONpCJzFZFIyJUfZeq79Lux3aHuUDNdwfS4ZLRZbELNZ+Fmg+DvQazM0eJyMC8OSqeu+dh9jSPvug7HUUN9Ubg28hCpAI+cFQs7XNFLp2dJ5xsQRlsUfc8Z6rlfXjq2w3TQcNaCybvRbQfG3L/iMExjcEkFUkiYqMiis3reWmQ+7EZ5PV5jkh4EsPPctJKMyd7tiLMxKx4VkTE+n+T05lf52HN4guB1wEdAFV9CmgWl6TpMZfS2UHAI/u+H6XW9XPBXmeNUZLU0O4nhHE6iKsg2E61V4CMtFnzaFRcRCBNzR0mQWPstHxPwjrZTmOeWKr52QZAa464aG4rjLG7kDthQi9KbQCofkIvmny5mrdnOy6nTf88SGcjVVURUQARmYC7tJKjSIwhTvZcVCtWTqfZjLqoapCqkqbWJbKIoJhBx+W5TiFePH3X4a6Vo01PibHpMUapVyZ7751+wlYnQkRYWwiojrFbOoxTNtt7ssiFU8ZuSFLDdsfKQps1j2vLxW58nCWqSmKUMDZ4ruKILdeeOlRPsQHcGGW7G9HqWZcoq80A19lzDdPqJ1R8h7WFYK6CGZ1nRuksfiZTQy2LyP8J/O/AjxabrOkwjzu4A88lSsxgEF0LXNYXK0PuPorBdwXfdfbJEKPEDLTf/gTcWY9L4Dn47uT3PBjVwVoJjG/2eXq7P/C0GpwhX7baETu9JHtXtPv5vOzMpjNyXQfX3csvEdn3elxa/YSttg22ZEPR7u143syi9XmOnKPwrsPPZ5RndecxRZuhRlng/n4R+XPALvBxwHep6juLS9J0mac9FmBHYK7IoCikxgwVhyILvj33ngRR8V0hMYpgF6DP0jCeOlUTLv2qiho7sjXZQmhqFG+MGNS+65CmaaZeMviuA4y/UO57e9LZpIBgTznDHnsVzXxrzW7h97DX45Lv+zko6x7+PM3K7jyjQx6A91YqLXJImRqW1kr2KvfAnBqQAt2Uj7Rcn3UOp+ogRMQFHgZuqOprD3z3tdgd4Teyj35QVd+Sffc1wN/LPv9uVf2J01z/OHKt/Tyx20to9+2O1GAo1sI06cepDV4fG+qBm8WbOF9hMnPyaGMidtZm7eTWD1U1cGgEHsODzzwimSpHyoXvXqmy3Y1pdWOe2QlZaRgbe3vMtC03AhoVz8pCT3BYeBZavZhumNKJUu5dq8+k0580jarH/Rt1umFKdUhSWw1c7tuos92ORg5zO0vixNCNUrbaEYFnvTpvtyOqgXUaeBDFhq/d6cYIcO96nbzbmLl0VkRa7LWpAVYN1VHVxRGv8U3AHwJHHf/vVPWvHbjmKvBG4MHs2o+IyDtUdWvEa47EPh//c8JizQNR3vfUJoKVUT70oquMMfA9M1XfpeK51u2F7G1cO28Yozx2q2vjYyjct1GnUfXs7ukwpR8ZwjiiUfUIsvbzyds9awYUuHulZuNhH7h3ERtpb6nmDwbop80d33NYKrjxbuaS0gGz3+E8CTzXYbG+P+/COB1IkHtRSm3dnet1IN9zWHSdTJZuF6iXjpHOCrBY91ms+0NP0D7Poi1uJ5ZSVW2q6mLWOdSALwL+5SgnF5HrWDchbxkzXZ8DvFNVN7MO4p3YjYETRVCSNB3EL5gHRIRubD2spApVf/qbxoYlfEXKS/ftWD1lDOzjSDMFTi4OyOMBiCOD3cD5DCInTPYkrMdtRBzkEfMlwT2MXC6cp/UidBRHEWV2ZVUmEjZ2EhwnF94vl2XvNXcO0PLfOiI47B2jCv0oATUzl84OUMsvYRvzUfgB4FuB41rjLxKR94vIz4nIvdln9wBPDB3zZPbZPkTk9SLysIg8fPPmzRGTtEe7n/LorZ4NjTnwWjl7rixUecH6AlXPwXUgStOTf1QARTaASWpo92JavSR7ndDqT1Yi67nCejOwi/eeM2hIHBGuLAYEnkPgOfSiZPCba8tVKp6D74qtgCcwz53EQc5TWk9Lo+Kx1NiLwT1LNyqpMTZGeTemGybEqX3di+5sa4b3hciwdp5MSZYadnsx7byOZMckqaHVS3h6O+RDz3QHETaLYBQz1F8ceutgTUP9Iw4f/t1rgedU9REReeiIw/4D8DZVDUXkrwA/AXzmianOUNUfAX4E4MEHHxw7l/SI17PGdYSXXVviZdeWpn7tm+0+f/RsizBJeWC1QdVxSYyy2gho1iZrAx7uFk7jC2ccDqZ6sR6wWL/Tw+tC1Rt75+04xInhdjukFxkWax4rC8Fcm0nmidQom62QdpjSqLisNSt3SLodR1hvVlhvTjbI02nZN/SZxDjo0N9Pp/UapVZ8/tDrBHgU+IIRfvcpwOtE5HOBKrAoIj+lql+ZH6Cqt4eOfwvwvdnrG8BDQ99dB941wjXHoll1uX+tijjOSCE0p8fsZI6PPL5FmClzokjBta/H2YcwCp7rDCLMiQjN2tCoakIkRrnVigDb0ExuYff0Nv/b7Yh25pZ82EleycnsduOB1Fhk2EY/W1nwUbiOtQz47p6bnKXGUVLwo+9BRPDcvYiEw7/N65HvOdSDYn2KjSKd/brTnFhVv53MRUg2s/iW4Y4i+/wuVX06e/s67EI4wK8CbxaRlez9Z7PnbmRiKILveXM1q8inkQdlcdNiqeZzqxMSp0onSnAqPr4jhHGK7052xD0pGeVRuCK4jgzydFyJ7EFyCap9HHqqhf+K79Dp23NE86bbnnOCXGo8lHd2vTELkKWKHfPNV6dxXDkflCnIxCRHS7CP+9y649nzeVYER9Z+EfnnHDO/UdW/cZoLisibgIdV9R3A3xCR12FnLJvA12bn3hSRfwC8N/vZm1R18zTXOw6dQ+lsK5PO7nRjXN9hqe5zreApdR5tTBFeff8qDz++zf94psVvP/Ysf+7Fa/zp+1dOvUt5llhPs3V6UYrjyEi70Ifz4qASqh+n9MKUrU7MykLAcsMfu1laaQQ0qx792BQetnWeCGPr4iPw3VN7A2hUPR640qAXpVR9hzAxfODpbTY7EYLwmS++gnPOoidEicnKVES94nJlaTxPs8MUvTwjR9mJs30OR1LEvoez8OCDD+rDDz881m9UIUrnq8NQVW7s9Pmnv/koIrDRCHjDZzyv0ABIT9zqEqV7KhKAODVcX61RCVx8Z77VPpPksZudwSav66u1fYqofZvbBmuMlyNfzsKz233afWs+WmsGLNXP7r342d0+73n8Nqqw2gh4zQPrmXoP5m1mcRwHN+WdRabuig3mNu7PReQRVX3wpOOOnFnMW2dQFN6cufsQEXb6dsNNnCp13y3cqVy+CW0Y37UhHb0pdRT5oCXfQZ6/njZJqoPBw0GfQpJ5NNU5NXccZH8+zm5vRZyavY1aE1o36sa281Gg4XtDsuDzhWSSZtXTmTWHcQvcvQ2jqaE2sC7KX4ZdqAZAVUdWLc0rRuero8h5yZUmn/vSK/zu41us1Hx6cZq5lCiGu1ZqbLUjUlXqgUs3TAkTwzPbfdaaQeHmkiixcZRRa2rohglGYaWRb06aXiNw10qVrU6MycxRh8VOPg+ziTgx9OOUKDEsNwLOGL78TFxZqrLVjujHKXFizW9n5fpynTAx3Nju4rpCagyOM/049Wcl3wvU6sUEnnMmc2+UQqVA/1CjrFj+NPDvsJvrvgH4GmD8TQ1zTCdMuN2KSFLDWjOYeJCfcXEd4aEXrvHQC9emcr1a4FLLAg+1+wmtTHHiOnKmBeGTiFPD7V0rhcy53Y5whJlJH+sV79AO4rzhOjJw/jiLotwNE263I+LEukK5slSZaJ3yXYeXXF3kJVdHdSQxnwj2WVU8Z+69445SK9ZU9cdE5JtU9d3Au0XkvSf+6hyQP5ubO+HATl1EkJ/TMRuzwXM7/cFC2cZSpdAZTasX7+sochaq3sT2dOyLhFekVGRkpiPzdByh6rhUCgiHOwo3d8NBEK/aiGFFJ8ccuzM5UAZzr7iNCQhIipxVwGg7uOPs/9Mi8nki8knAanFJmh65GSoP8iNCIcFYxkuTZn+zCTpfq+zlRf+QnaaTwujevoe8fOfXnYSk1KgOpvipMYM8NQW4FRmF3JOvMZkKb0ppmNXAp17xpl6nZl13jmPw/PVw1zaTeE7RHAQ/+m4RWQL+FvDPsQ4B31BckqZHnrFXlirUKi5RbGjM2LtqqxfT6SeEWdD5aUxNVZVulKIKVxarmAXbyFYOcaI3KVrdmN1egufAxmLV2rGzDqpyipFoatRKZMUuon7kVpvbnZCtXswn3bOMJw67vYSVhkc1sM4Ep7kgmqZKO7QBl8Ca2Sq+eyE8wB7GxmKFlYZPnOqhzhjHIczWXnJlWhSn1r179trzHCqeY2eq/YQ4qzvuHJl1wtjQixK2OjHLdZ/lEb0UqyphYkgSQ3BCeSm6azxSOjs4QGRDVed+jeK00tnYwOO3uoRZwKFrK1XqM9S/D8szgaks8A7kogr3rNWmYroYlgzC2e4zTgyP3+ra8+6/Cs+7sjC43mO3uoO8fWCjUUj0v6PI69ntVsR2107Wl+s+a81gTsye88lOJxrswh824A2/rgUud61UB8dMokxNmn0S2TFk17dbIdsdW16WTigvRUtnR+l6/5uI/JqIfP3QjuoLgyPWLUS+Qc+bYgNyGLk8M389DdIj7r8IT7D5OXN3DXmlOYtpJh0KCDOM6zg4wqBTGJxeKdyd80Ek80ybDt1jkeKBi0Ji9A6XSgdfO44NFnawTM2+o8g9G+95ixVyz7KjpS0duv+T2qaZS2dV9cUi8irgy4C/KyIfBN6uqj9VXLKmg1EbKe/ulRrbnYgoMSRGudO93PToRym9KLXO+xb8zP11sde8e7XGzd2QMDY8tdVnqe7huQ7dfpKpg84+0zKqRLGh3U8QgdWFvRFSP0ppZ9JZO3KCcSp6xXO4slRhpxsPPMl2+gme65Cke6Fhvcz9c+A5NvKfP57UchL7P9YWAjxH6Ebpvo1YJYdjZb9CO0yoB9YxdzdMqVfs7LcTpviu7I3c52iWFiWGJFXaWT1qjFmPVJW1ZgXPETpRavf4cHR5KVo6e6IZat/BIuvAPwb+sqrOlaj5NGao1MzPPovUKM/t9OmEKalRnmp1aUUJzarHJz+wRq3guBYmm1r8z2d3+chmh9QoL1pv8qIrC4M4EGchL2f9OEUQKr5DnCo3d0N6UYrnCBtLlcwECKdtQlXtOVv9BFeEu1aq+J6DMUoYm6EF/PHOv9uN2cz2oqwvBDb4zBw1TJeBODE8l5WXRuYaw3WEfpzy+ze2eK4VslIPeNX9q1S82TZPu92Y2+2I1CiNisvGYmXketSLUm7uhkSJYanusdasjLy+dprO4sw7uIdOtAh8IXZm8QLgF4FXjZec+cR1bJMUzUGH0Q0TupmMtBMntKIEBSquS+AWX/AdERD42FZ3ICO+f60+sQX2vJJUhxav233r2x+sdHASa0VRYtjN9ong7AUwclw5U6d3qxUOJMXnIVzneWOfxPkIWllIXLC763OrzLOtPjfbIYotX/4ceJC+3Q7Jw6qvNytj1aOtzMoBUPFHF2LMg3T2D4BXYJ35vVhVv01VHykuSdPD6Hx0FJAViqz0Vz3rbM0RaEfxPjt3UahaielawxrhBFsJJy01HW4QrCLJXiuMJxPgyXcdgmwtIE2tkmR4neS0LFQzKSjQC2crr75I6EDuqieWtaPKy3ItwHccXBHa/bjQAECj0qjslZfumNLhRj77ZTz5etHS2VHUUKLZQSJyTVWfKS45p+csZiijSqefZFNGb7Dzddr0o4QnN21cKc+BxYZPs+oXujEu53YnZLMT8dHbbRYrPk3fo+Z7rC0ELDb8iclMVZV+bAhjK5GteEKU6sQVWJutkDAxhInhruUqlTHXJw4j37dxUeWusyBJDZ0wZacb47vCteXqseUgl5IeLC+qym4/plHxCnW6eRzG2HbEqFKrWLc5cWrdreR1WFWtqTk1VAOXOLVR8OoVb1+5Mqp2XW3MejFTM5Tu701+BXjleEmZXxyxf0/c6hFlzvTstvvpe1nthglPb9mOouo73L1aG2xqmgar9YCVWkAgNvyoiHBP7nV1gtfZ7ljbvwKLNY/qYoWqP/nKvbIQHNgse/alZHdEN+clo+M5wmLNY7E22pqciFA9pOMXEZZqs5SmwOO3u6TZrnVt7Ul8m1Ufz7Ez26e2+na2kP0mP+aqI/hD7Y4jMvYAx5UZq6EOcKFqiqLsdmOiZM8rpjOjxmDYF31eYKbZYe33iJEVWGf480wJNHzwKRgee4jIiU34aT2nDkuQjzn5PhnmPOnyLw1DXlft2/Ob/wfj4ww6hKFbMgfK3KDdmcB9F910jdtZ/GghqZgRrV7Cs7tRpmG26wbGnM22fVoaFZery1V2uzHVwMkK3vQqUBgbosTgOLZQBL6bjZKUKE5p9RPqFdc6WTzDdZYXAlzX4eZuyG7X2pevLFYGDfXw2kI3TOiEKcYYNhartjIcc/FxG5x+bAiTlG6Yslz3qQ6UWCXTIjWGfmxo9xJcR87lJsW83N2zWmOnG5Ma6wki33k93KbcldXxMLHHJKmhPzRYPQuxoVDvwqOooT4L654cYLxFgTnHcx1QWKp5rDYrWUbPpqCKCAtVj4WqR2qUZ7b79CKrIb97tVb4ukXgOfieQydMaPdTktDKExcDF9+1+y58bxKjH2Gp7rNY84gTMyjZUWJ4dsfu9aj6DnetWBcgvutgNAsve8TljVGe2enTDa27j+trNYIRpJMV3+7JaPUSbmz1EeDacnUi+0pKRsMRoR7YuCnnMR7F7VZod+SrdRu0PtTZNQ853nMdVmfkUfmsHBdW9V7g3wMtIFc/fZGI9IAvAL5KVd9SfBKLo1FxedG1OrFhrmzRvSilPyQRnEba8iAs3SEvsPUhP1mT8hOVzxxEhMB3B++7YUqYbXoJPCfbjCj4npyoZMpDU4LNr+GO9bjf5qaqfnZdhbKjmDJ5Xo+7kDsvbHXiwev6mH7lzqrQG0YAf4Y7uP8F8M9U9cf3JUrkq4Hfwdatc91Z9KKUj97ssdLwWW4EZ45UNQlUNVvoynxXZdNYp2DXEJp56/Q8IYxM1oAng9gek9iUZ31xGVuwXWffeWvZ6DIxWQzsff6xjr+2n+3aDhNDnFqzRiVbNE9TJZ9kHHWehapLp287m16UXqrY2PPCec3vZs2j3bN7onphMpKr8dxslaQ6cNFx5vqFNUMFMzJDveRgRwGgqv9WRN7MBVBF5c4DtzsxjiMs1fxZJ4l2327OCzyHjcXKVEZcrX7Cr3/oJs/s9lmte3zcRpOrC5V9G+jOilHlqZ0eT273EBE++YH9gZ0qvsv9G/VBBerHdi2h4js0qt6xJgrXEe5dr7PdiQjjlI/dbNOKEgRlvV4dqLqO4tpyjTRTGMzTDPOik0vWw9hQr7jnspO+ulRlvTle2YlTpRcltLp2HXBlIcCodQsSxYZG1TuVp96id5cc11kcmlIRcYCeqj5XTJKmR7PqsbpgZrq34iALVc+OTrL306g7/+jdH6UTpRijfPkn3EXNdWn1U5Ybk5squ47D9eU69yzVQA43LeVmp3Y/4dntvo2vHLgsVEfTYSzVfTqR8J4nNzEKFdfhT96/NtKiX9lJTJ+bu+FgVO65MpFwq7Ng3LITuIJf81ms+YMG/tltu+amQODJqeTkXsHS2eNS9Msi8qMi0sg/yF7/EHa/xUiIiCsivyciv3zId98sIh8UkfeLyH8RkfuHvktF5Pezv3eMer1xcERYb1YIshH0PIxqrOdMGUpP8WnK1azWEVtxJgGxJz/R3Ldva88JScl3Z+959hy+3t7Pc6+28xYU51Iz+mMGZhe4ajLsldM9Se1eXd/P6epf0c3XcUO2bwX+IfCYiDyWfXYf8BPAd4xxjW8C/hAbNOkgvwc8qKpdEflG4HuBv5R911PVV4xxnbExCmneUGYPcdaKjCgxeO50lSFv+Izn8Rsfuc1mxzo+84dGeZM0Q3X6Ce2+9dt0NYvJfNj5F6oejiO0ewm1wD02Gmo3TOhGhihOubJUpeZ7fNoLNnhss0vNd4hTY+WJsaEXpVnQIaeUyM4BG4sVaoFrHUmeMLPPAxuBVR0d3sjOL2FiCKNcgu6xXN9v8r66VKXVT+hH6ald1xctnR3F3UcNeGH29iOq2h355CLXsZ3L/wN8s6q+9phjPwn4QVX9lOx9W1UXRr3Wadx9GLX+VDbbEVudCFVYXwhYakzfo+hmN+It//0JnmuFrDcC/vqnPUAjKNbT7DB5OYhTg+c6E1/sz8/f7ic8uxMCUA9crq1Uz9Qx5gvzxuylO68tqsrTW326mbLs2nKVRmWynWDJdMjdZOTmyarvcPdKbWabaMdlr37pwFV+EWVw1u4+esAHxrv8gB/AzlAOkxwf5OuB/zT0vioiDwMJ8D2q+kunTMOROAKBY90J533mrDyKfmyzx812RKqwsRBQmXJIyPyec5USTFjal51n2AFcxd/vSuQ01xMRXAFH7lwQTFIddBSuI2PHEzivTPK5zQsiss/Tgg29O9MkAaPn9V79Km6gEhQsnS2sRRKR1wLPjeKhVkS+EngQ+L6hj+/PeruvAH5ARF5wyO9eLyIPi8jDN2+OH/k19zq71PAHtu5OfzYeRZ+/VueepWrm7dXGY5iFfV2yxWdVJcpku5PMj4Wqj+/mHYet/Jo5TYvT068pHKyAmm3ky69lMgd0FxlV65QuyjztFuqCdAY0KnsxqKMknent5d5yo8SMtZZSZCcemRl7nT31iUX+IfBV2JlBFbtm8Quq+pUHjvss4J8Dn3GUwkpEfhz4ZVX9uaOud9bgR/k+gFlNa1WVZ1t93n9jh2bgcn25wd2ZiWaao8Q4NdzaDelGtjJ6rlDzXRbr/kTVKqlR64K9n3Bz18aKyF2eTGK9phva2AdhbFht+gSug3MGb6RxatjtxqRGadYmmxeTopPFe4gSw5Wl0YPtzBtGlXYvoRsl1AMXEPu6Yj0cqJ5dvdaLUnYzT7eLdX/suC39KB3Iu5fqPvWKS5IqO70YY5TFun+ow8OimXXwoy8Efl1Vd7L3y8BDJ5mFVPXbgW/PfvMQ8C2HdBSfBPww8OeHOwqxsb67qhqKjc73KdjF74niiPXUmOqIjucK5JlWn0ce3yRVWKh41h5bsBTuMG7c7g2CH4E15bTShKW6P1HzhusI252I2y3rm6tZ89hYHD0i2EnUAndfg37W097Y7JFkaohJ58WkqFdc6pXJ3fOs2GxH7HRiFGj398yWruPQrJ59QBfGKU9t9lDszHNlYXxvtVXfoeI7LGUL1SLw5GZvsF9nuRFMvYwUvYN7lO70jXlHAaCq28AbT3tBEXmTiLwue/t9wALwswcksi8FHhaRPwD+K3bN4oOnveZ5wBUZGF90SmwFhVMAACAASURBVKanYdlpbrYYlpvCsPR08mnaV5H07CLh3DSw51AwHwScXYI8fIZ5Ne7k9zpN2fWZOVAGVfXIVE+qDMpwXTvtKQ/J6311ZwY2sqIvOYoa6v2q+gkHPvuAqn58oSkbk7OaoYyxTfSsNmepKtu9iPd+bJPlesCVhSp3r9QKldH2woRebGj1YlYXAhaqHkZhtxeTpIbFmk8YWxcazZo/8aA/qkovSmn1Ehaq7ti+dQ7S6tkwrb045a7l6kjOBEclNXaHbZ4v87KJ87xjgx8l7PYSPMcGPwLohAnPboeDRn2h6tKs+hPz3RUlhu1ORMVzaNb8iZifU6O0MjNUc0ZlZKZmKOwI/x9jfUUB/FX2HAuea/JMvdUK2ckcgq01A5bq05fOPrnV45t/5g/Y6kSsNSr8vc99CY9nMrt712q4BaijqoFLJXBZrHq42ULwZjtit2vzIq9IReVFPzY8s923PqNSQ63inWksnO9+N0Ztp3/cBo0xcR0ZmBxKJofrCM2az8LQQEFEaFQ8nnfVo5NJrTv9lNQotUrtzPOl3KtznBjaYhWQkwjx5TrCcmN2AZiKbrFGaYH+OhAB/y77C7EdxrnHEXCwI+tcg3PW0e1p+aNnWmx2IqJUubpUHSy4+Z6DFDTbyXeQuu7edHo4L2oF5MXwTLYXpZgsYEzFd89U2HP7sCOyt7B7Xo3254hxTUMHj8+fmePYv+EOwxEZSK3t3oqzlZGcODHEmQzXcx28c7JX4ziEOVizUNWOqv4dVX0w+/t2Ve0Ul6Tp0QlTPvRMlzQzRQlWRTML6eyfvH+FV963giOw043oZfsDEmMGi2ZFkctljWqmPrHcbkUkqZlIfuT26DAxmVdZZSFzmAaZd91TXEZVM0+z6Z7Nu+wkCicvL73IjvhPKiOD5x9nDXV2/N7nKVH2DIdp1vyBB2ErCT572gPfoVG1HU9qrGz7vKMUL509Lp7FD6jq3xSR/8Aha3qq+rpDfnauyNUtYZqyG8YgsNiYzcxipe7zna99KU9v9nBdh4Wqx0Yz2DfaKpI4NUSJbWjXF3xc16Eb2rgajREd+R2HYuWGnX4CIqw3AwLP4fpa/UwNfJi58YgSg9+cTuyPEohSQ5xoti+JfSqsw0iN3ZPQ7icE3p6KyGQdSDtMcEVYXQhIjQ133AkTFqoe9wzi0Z/8bDXz3rrbS6j6DssNH/eAZNoR4dpybeKhXMM4zSLlKY2Km633KcsNn3rF1qE4Nex0Y8LYsFyf3BrMNDiuFfjJ7P/3TyMhs6BZc7m2HPCbH71NmE0vXnZtcSaj006YcnMnxHUd6oE78H8zLe1N4Dn4LvvcYeTeXieRF44I9Yp3qM+ps5y/GriDkadMMb8uOxXPIThQXo7DcwXPvfP5u45DveLs+3wrjz4H1AIdy/VMJ0x5bifc56zvKCZdx29s9gaz485QELH15p6M9pmt/mBz6OopJLvHMbPgR0M7r1+hqv90+DsR+Sbg3cUlazoIQrMWUA9ckjDbtZzOZko6PCA2++SD0+q07txnUkSHWfw5z8co7fwz7r6kvbWIYYbNTmo/wHHs0QYlMdb3l8NoA7jhmeVxMtwicB1Bs0U4Za8kpqrk0ggn8wuFQpoa28JPCKPgTE7TcQejSGffp6qvPPDZ76nqJxWTpNNxFumsMcrHNjtsdSNefm2R2hQd+A1jp+kxVX86gWCSrGMcd/fqaTCqWdS6/WY166JCcScoEVa1jYwDhajISiZHkljp7Ha2m/qulRoA3Sjhj59tc7PTp+a7fNoLNkauD3FiaBVQj06qL7ln5dTY9bh+LjuveoPfqCrd0JpNF2revhDAk2Am0lkR+XKsX6bnyf54Ek1gc7zkzCd5pv7c+5/hvU9s44hdJHrN/SszsSMGnsPqwnSCud/Y7A3ifF9bqVIvsHNq9WKe2wlBMk+zy1VEbNjWp7f7AFQ9h7tXa2dOQ2qUJ253SY0dVd63Xp9KZ1hyOlxXaNZ9mjV/MBQXEeqBxydcX0J0aaj1Gx6vH40/4Xqkqjy52SPKNmXds1o7NCa9I1YGnLNwSLkTERpVj8Yd35ydolus44bQvw08DawD/2jo8xbw/iITNS0cAdTw/z/TGri4eOF6Yy4WnIpeN8nVVgC1I8KnTioNvSjNTAxWIpvTj/fULcGE/OgkqSFNsw2WrpQL3nNMXr6sVWZ/WRMRvEyll7+fFYoVUuQc1lHMA8EpZhXjcNyaxWPAY8Brirv8bGn3Ex671edP37fEe57cpRunfOhmh41GMLPCEKeGZ3ZDWmHCS64uTDyuRM5GM2CrE2NU6UYJjaE9Fflu9l6UUPHdzP/+6dOwVPeJU6UfpcSpGeyVa1Y9+nFKL0xJUuu90z3jveZKm91ejGBNe5UZOHQrORqTdQC9MMV15NjGdx4aZUeEtYVgsOje+1/tnXmQJNdd5z+/zKqsq7v67hnNpZEsyYcEyGYwCzZeLK+FjY0Fy2HZwXIErCCWUwvhgPWCFxME1y7XsuziA2MvYCOMDQLM2gbbgCN8aGRsyzosyfJ4NKMZzUz39FVnHr/942VWV7emp6/Kyqru94nomaysrKyXWS/z5fu97/t922FH3TRItMKdhaG2ytXCUB9X1ReLyDJrJSYCqKpeyfluqAjjXu1zZ0f5pmdNZWoYr6os1H0ev1DjqaUmTs7hhulyz2OaCV7excuZJ35XVhuDph9ycalFy4/wcg7To0JulxlWC3mXw5MmFt0OIh69sMy5pQaHx0pcPz1CznWIImWh3malGVL2XCZGnil57EZVWWoELNV9vLzD1IhnzI9EmK4WmK72J5xn2T6qxO6FxhWusG6QN4yUy7U29VbIaNFlrGwk5GGkLCZ1pOAycQVZbFpMjHg7Sji4l7haz+LF8f9bMS4aSqpFlyMTBVzXzTydc60VMr/cZqKU59BYgYPjpVRDKEnWTQBvvNgJCZy73OxMAjw4XiTv9va8fOnSMo9dXEGBA5F2cvIsNwMur5jZ44Wcs+lgd9M3qdSTYxgWxzSLUQ25zqo3xVrJs5iss/FTfKSrJkdLDZ/5FbO+mN+8juw30g5DXbVZFhFXRB5J7+uzpRlEfObsApfrbcJtGJikQXfDEEXmR0/T/Mh1Taw4imdAh5ESRdpJfSCsqj96SSl2OHNFaASr4ybd37uVGetu1/ZRhr+bZfuomroGpnZHsTw2UvNePq6bAEEcElWN7UjZeh3Zb4RZzeAGUNVQRL4oIsdU9XR6xciGp5dbXKq1ubAyx80Hq1w/NZJZOqGS53J0usyTl+q0gohTF2ocmymTk3S62ddOl/nKXJ35lTYfeuQ8N0xVmCgVaAURlYJLpZAzuXh6dELCyLi4HZuoMDNS5MJKi9mRVf+KSjHHsZky9VZIcQsDiF7O4fhshVozMPmzelJKSz/wg4h6O2Sh5pNzzeTPhZqP4wizYwWqpTxFz+XpxSbLjYCWH3J0qsxovL7eCjMNGQ8qoW4tM+xO2cq+J4AHReTTQCcn1F5I9zFVMRPynj1b5ZpqKdO8cy0/5EyXIcuRqXI8UW9rcsHtIiJcO1XmyESJA5cLNP2IWivk4HjRpCCgd4OLS/U2F5faIFDxjBvedVPPFA/mXYex8tYbR9cxLmeW4SKfc6i6DtVSPu5Bm7+5lTZn5xqMFHPMjhU4OlUGumu/bruO7CfSjsRupbH4hXSLkB0jnstLnzVDRPYSy3YQdSJO+ZxjjFRSbr1EhJwr+KF2nvC7Y8G9ks42Y69ttHcSWcvwkrhSJmFfx5FOCgyTgdg0Bk4snRXzoWwK22PSlMRnlu5DRG4ADqjqP61b/2LM/IuhZ6UZcnquSdFzO4ntsuraVgo5qmWTZKzRClms+YxX8n25UA6MFbm01CKIlAuLLSZHPFxXqDcDip676/MyUfGMRLcVEkar0lnL/sSMUSi1VkjOEcoFl8kRjyhSE56q+7iOCU/tlVBTkpm31jJy9EIK95rMpLPA7xB7aK9jMX7v21IpUR9RADG66bPzDVzH+PGOZlBJI5RL9Ran5mqU8g6VUi6lANQzSVKFh5GpzJEqjko86L/7/edzDgfHS7vf0QYksuOluk/edZiuFnru6mfpLarxfJ64gns5h2smigRRksl2+Aaw662A+ZU2kcJkJU9l3X0kUiNeiYZ0cP5qjcUBVX1g/UpVfUBEjqdWoj4yWnQ5NlXk9FzT/JChUswoTHJ+qcnjl5aJFCY8j5nRQv+kgarGBMZlTS9iInb92n3DuSqLTIOWHzG/3I6/JSLn7o2n0Z2R7rnuBYnR0XjFjDd1Gx7l3VVHwmfWu0E8ttVHuvMLzU7W2eK6AXjXMRl285WNjm33ZCmdHb/Ke+k9JvYVwcuZtMlmQJeOM1e/qcTJC3OO0PSDvlwOJomfMR0y4SFdNUJKJIts3w2te/+RaiyJ3Pl+NiPnCoj5/aLEem8f0n2uoxRl170icWdMSOqLdr1OeGY9yvbYkmskis+1qsYPWqYetoP1pmGrDWJaUYsgQ+nsSRH5j6r61u6VIvLD7BEP7uS+cs14kUsrLRp+lJkkb6Ls8YrnXsOF5SZjJa8vk8wu1dpcWmny2MUVnn94nMMTZQRjUtTwQ+ZXfCZH8oxXvB01XmFkjGgWam1c1+HIZDrPGDnX4brZCvVWSN6VfTtBb7nhU4sNq45OlXBy6fSSE8dDk+tr+7H3dhARRhFezl07v0jNMSzU2riOw5Gp1fqyVPdZapjw1NHpMrKDGpkYLbmye6FFox1Sb5nxlQNjhY5JU8uPCFUzuY+kHd3aMEW5iBwA3o/x304ahxOAB3yHqp7f0heIuMBJ4KyqvnrdewXgXcDXAnPAa1X1VPzezwM/BITAT6rqB6/2PTtJUa4K7RA+ceoSl1aMtPMFRya4plrcMwNrV0NVCSOjhFLM/0nPYv0z0U7OR/d+kk/vh/OaFcn5TuZDm/tw78/3k3N1o97DPGht58Z4canFUt24Uk6U80yMrOZhu1p9SSZerq7vfrU5fhBxeq4OmAmhx2bKuwrzajyel0h/+6Fe3AxHdqaI2nWKclV9GvhGEXkpcEu8+u9U9SPbKwo/BTwMXCmX1A8Bl1X1BhG5E/h14LUi8jzgTuBm4BDwDyJyk6r2NEakGvHRTzzMeadAsWQaiNGMEoRlkV0zkc6qKk7X5L9OJlDVbV+U6/ef7Cd5vRHd2+zmXKz9bL8kAoOBOd/mJpbmeJff5YW9XSFBO+iSUa/7bFJfzN0Xun+7jmuksqPAfNBlSuT2YEyrW/47KD3ZXMrS2U1/aVX9qKr+z/hvWw2FiBwBXgW8bYNN7gDeGS+/F3iZmKv8DuA9qtpS1S8DjwMv3M53b4W/+6cv8Po3vI3f/K17OHv6PBXPZbHh9/prNsUPIpYbxgAmi1jz1TN+7rz2Rar4QWSUSg1/wzELP4hYqvss1n1q7YDT8zWeuLTSiQVvFT+MqLdCLtfasVRxx0UfYtLPcXYw7k14OYeWH23r+2aqBUaKLnlXCKIN0slsWO9kx3fDYt5hctQj75oHpF6lC8m6N9FNO8ww3UcP+B3gDRjDpCtxGHgSQFUDEVkEpuL1n+za7ky8rqd4eRdVOH5gjNtvOcoN185m8uM7juBrxL1fuMDjczWunSjxuucfYrTYn9nJ//ToRd7x8VMstwJ+6EXHueVQlTCCqdE8o8X8js+JYI4t56y64NXbAQ+eX+LicpND4yVuOThmtnEdkxfIcSjmXYLYvGg73+xIkqRue+k/FuttFmomieHsaIFysbeXRcsPmVtu0/RDxuLQyzAnwSsXcjtO0e2lLKPeCBFhouJ1FH5bYaFm6oUIzFYLlAYwLXk/Se3oReTVwAVVvV9Evjml77gLuAvg2LFj2/787S96Hh98609y7PABxqvlXhdvy7iOcGq+wUMXVowdqCN9lfD+6gceoZ645lWLtAPzeFLydjffRERwxczjSF5/eb7GU4sNADzX6WT7LRfczjazo8XO8nZwHQfX2/6g68Wldme5VOj9eZ9baXfOr81jNRyEkXJp2dQLwUhhNyXj2aZpTsiDLYShdsGLgNeIyCngPcBtIvIn67Y5CxwFEJEcMIYZ6O6sjzkSr1uDqr5FVU+o6omZmZltF7AdKGPTs7j5/LZDHr3mYLWI6wieK8zV2n2VCN58qEoh51DKu3xl3gwCihhV1G7OSbcEN3k9XszjiJEILzdXQ37dksLdygu3+9lCfAMXoPUMyePuKeVXpdnNjKTZlu3hCHF6fvPaDzeuE0k9D7tktFmQdhhqQzVUT7/E9Cx+9gpqqB8DvkpVfzQe4P73qvo9InIz8GeYcYpDwD8CN15tgHsnaqiFus/ZeeOJMJVIRDN8Mji/2OQ9J89w/WSZWw5VuXa6jJuS+VEUKXO1FpHC1IjHmcsNVpoBNx+q4odGJbWV7K9Xo9YKWKz5NNohhbzD4dhj2w8j5uttJsteauZO26Xlhx0vjTTqQBgpTd9kSx3mENROUFUWGm2aQcREyUul17xqpuT07DfUWGorstYOWFWNVbCanmizHdBom7G56VGPauzD3fBDNDK9kn7lnttJ72LXaqi0EJE3AydV9V7g7cD/FZHHgXmMAgpVfVBE7gEeAgLgx3qthAKoFFzKBZexcj6zmdsJn3tygZ+553OICDfMjvDKrz4YqyzSUfT846NP0479Kl7yrBmunSx3LjAvt/vvm1tudcYBxsp5pkZXY8V51+FAHGoaFNK2XnUdobJPY973P3mZ80tNAL7q0BjHJso9bZDrrYBzl83+C57D4YnejImIyBXDT6cu1jtS3mNTZUpejmIeJke8zpP9mflGJznoNRPbkxfvlLQTF/Sl9qrqx4CPxcu/2LW+CXz3Bp/5FeBX0ixXzhEmKrnYjjNbZcN8rY2Iyb45WswRqSIpeVkANIOwM4mnkFv71AS7PxdBV/gpMbPZ6j6zkBGnyV47nu3S8MPOzOxSGr2KLiMlVyR1wXS3kspNDJkcWSOjDcJVNV6uT71nN2vp7F5mqRFydr7FU5ebcbcyuzGLF984zX/65mdRLec5tdDgLz57jlaQXpledN000xWPSt7lyxdXWGkG1FsBF5dbux6rABPaGq/kKeQco2/f4u5afsjlms/cSnsoUlZcjTBSmu2QueUWi/WNpcN7nRccneD4ZIXxUp5WCu6LI8UcU6MehZxDPlbVpcnhyRLlgkvJc2kHYecO3f0wcGiiyEjRpZh3CMLtyYt3yp4Ys+gHOxmzWGkGfOVSc3XWqJib3Fh553LR3RKEERdWWiy3Qm6cqfTUhGg9yeBzoxVQiGPpjbZxqsvKk1xVjf9FJ2UCDOvEumRGcqMVknMl0xT4u2Wx1ma+Zho8M1di76QP30vsqTGLQWKkmOP62SKPn68jjoOq0ZBneRHkXIdDY6VUTVISki5zdyrlSiFbu0oRoeS5fTn+tElmJJczPqe94OLyqrx4K9fIXvj9ho1hls4OPA996Rw3fct/5a8+dB++HwDQaA9GLv1+XGhJls8wlvsN0gU+KOXoBXvhWDpWu1xdUt3JNByt1ilLfxj2GdwDzee+eIblepPf+MN7WW61ef1rvoF8Lrskgu0gotEOyLkO5T6oJ0x22YjFWpvpqsdIn2aMW4aPQxMl/NDkhLpaPqiWH9Fom5QrXs5hpJij7Ln4oRJEEWUvR34b+aQS+WrLD43D3C7l3L0gipSVVgCqVIr5zC2ZE9Julvd1Y3H7Nz6XO277Gp77dbdw+PAMj16qMTVaMpOo+lwhE/mfAkXPoeylnxKh6LkUPJeJSn6Ih5Et/WIrc2IKeYdC3qHeCmj4EU1/dRa0AgfGTG6mrV5fi3WfudjYqlrKMZMv7PwAesSpizXzBC8mJGcmbmffYOwJ6eygMl6t8Ma7v5vPPbXY8bbwMnJZ685rZnKHJsO66QkBO1k+U/uGq7PfJaV7kdWU4mvXJy+3OyGx24J0UCYzdoqka6+hrHF2nmdxa/tPb9eDz9nFBp+PG4pS3mF2pEA76I/MbT2VgsvsmHlqarRDnl5opp45tR8x5Y32X2sFXFxqcXG5NRDOZ3uBQRojODhRZLySp+y5zFQLjBbN/IoLSy0Wau0tl3O84jE16lHy3Didfpql3hpHp0qMlnKMFF2CMMo0H1Q3foZOeXueUt7tPLc3/YhW0GJ2tMj0SP8HesNIWWqYQXZHoFrKp/qkML/cYqHuowqzY72XQjbaIRcWm/ihMlrMMTtWQERYbPj865nLLDZ9psoeLzgy0bPv3K+EkfL0QpN62zgFHposZZ5GJe86TI+uhoyqpRyzY2ZMYzt+Esar22O8kkYpd0Yh73JgLNuMD1mwr3sWMyMFXvW8g+aJBdO9PDBayKRn0WgbO0yAkudS6sgt0ynLfM3vhN4qKciFF2vtTvK1orc6KHnmco3FOIHgVMXrSoMwGE9nw0izHdKI604uzuQ7SCQqOxGhkDcT52zosfd4dgZ3ekQKjuNyfLKCI+aJ/kInLNJfCnkXN06L8Uyz994zUsx1pJD1Hs5eT6STSU4dARrtqLN+aqQQ+1vAQsNP3Td4P1DIO2bQGGMkFQ3ISU1se1ux5/YgSbP3IjYMlSLJiX32bJUDI0UafsjsaDbS2bwrHKgWWGwElL30n7ymY9cwjbSncsQwUuqtgJVmwEjBZWLEw8uZCY8rTZ8wgFsPjlMdyTOyS78MiyHnOlw7U8EPTIhnUAaCV1oBc7UWX7lc48BIkZsOVNf0H1WVWiukHYRxMj7b49gNVjqbIo4Yudnp+YbxFQbCSjaT0+rtkPOxdDYMHUZL6c55OH2pbjwzgLERr2fHnHOE0VJ+TfkTv+LRUp6RUn5VgWVvDD1lO/MX+sFIIcdIIcexycqqrXYXF5daLDcDVGF6VCjmB6v8w0YuZTXUvm4sErpDMJndvrqls326iXZyYvVyp1eREg6SzNCyRVTXPLFunqtrrYKvk9vsCh+JukRwtl7snrRvG/u6sYgUQjVZJBfrvklgRzZPvOWCy8GJIos1n5LnEGm6k2yOTpVZbPjG4SvSvqVRtgwXjXj2dK0ZMD7ixZkFNt6+HUS02hHLTZ+S51Ite7iymlSxO0Q2Wy2w7LmdrAWW3eFH6c612NeNRXJS//mJeT706CWCUPm2m2d5yfWTfW8wGn7IfafnWWr6VLwcL37WNK6Tnjwvn1srbbRYrkQxnpFd8twthbnyrkO+5ND0Q+ZWfOZWfHKOeTCLFMYreaZGjCOl4whj5TxjZZtmZhjY142FI0Zu9i9PzNMKTJ795x0YyaRnMV9vs9wyznIjhRw5p39PWlalYtmIJHS4lfTq3fWo3l41tgy6LCwqfch5tl+x0tkUiRTaEXzT9ZMUckaf/vDTK5lIZyfLHqMFM/i70goIot6bxKwnkbO2gyhTo3nL4HO1G3wyczyR7aoqY2UzqRQg59BZrmVsMraXaVvpbHokJ/a2G6f5t8+aohVElK/gudsPSnmXrzs2yfxKm5GCm3rPwg8iLiw2aQYRrgjTVa+nHtFRpKw0A5p+QMkzydaafkC5kM/cM8PSW9pBRNMPWWkEjJTyVEs5xivGRCxSYz1qHkwYuAmDvcQPI5bqZhywWs6n7uveb/Z1Y5FIZ8O4EmfVUADUWyHnF5odH+G00xs8OVfvTIg7Ml2KnfF6t/9Lyy2WGwGKsa9NcB2HSmFvXUT7nULewcs5VGO5dLc0OhFpdC/vVc7MNTr+3GOV3snRt0rehqHSI4giHnhqgaWmHyfty7B73PUjJ+b2aSfYe2a96l1N2/AisRGIPYh00nkYmezebBWScNtqwsa1lXlt0pr+V/S0b1+p9SxEpAj8M1CIv+e9qvqmddv8NvDS+GUZmFXV8fi9EHggfu+0qr6m12V8arHJl+ZqfHm+zs0HqxybrGSm9y57LocnS5y73KDpR5yda3BosoSb0oV3bLpsnvxVU8nHPx1nCm20AyqFHJEqjXZIyfYqLENKI87BVWuFTI14sV3u6vtHp8ssN3yCyKQn7XeoNVBwdTilsy3gNlVdEZE88HER+XtV/WSygarenSyLyE8Az+/6fENVb02xfFSLORwRbpoZ5ehEdg0FmJt10XM5Pmtmu0aq8aBgOn4WOddhYsTr+X4TRISRYo6R4moVs058lmGm5LkUPZdqScldIabmxhlysyLt+1dqjYWaftpK/DIf/12to/Q64E1Xeb/nTJQ97rjlGlqhDsTA22p2TpABcd+yWNJi2CTbiYzYyQ1mmYd6zEJEXBH5LHAB+LCqfmqD7a4FrgM+0rW6KCInReSTIvLtaZQvUghUcPs4p+FKaCxfbXWyzQ7XRTSsqCpRpDTagZUO95Ek5t/yI5OTbUjO+6DXl6GWzqpqCNwqIuPA+0XkFlX9whU2vRMzphF2rbtWVc+KyPXAR0TkAVX9UveHROQu4C6AY8eO7aB85n8/iFis+/hhxFg5T7mHEtKt0GibdApPLtRpRSGT5QLXT1fIu+nF91t+yGLdJ1JlvOx1UorvJ1qBSWVRb4VMVIzXgmWVlh92DLImKtuTggZhxELdp+1HjJfzHX+WIIy4XGuzVA8QgbFynskUw6G9xNSXiHor2Jf1pS9Hq6oLwEeBV2ywyZ3Au9d95mz8/xPAx1g7npFs8xZVPaGqJ2ZmZrZdLkdMpsanLjdYqPvUWmFHE95PyoUcPhFnlxtcWGkzX2/jSLo/zZn5BkuNgJVmGNtVDtZTUj8oxnLPg+PFOE171iUaLJ6ca7DcMOnm866zrcfWc5ebLNR86u0Q6QrxXlhssVg3kuqZaoHJOPXHMGDqS25g68vQhqFEZCbuUSAiJeDlwCNX2O45wATwia51EyJSiJengRcBD6VRzgiT3CyRvYUZGMeoKm7cOAgQqqZeEbuPeR+2EwZZJ/m0uU/X4MiqHDSRc29GEmLqrr/d15TTNZ9nUEyarkaS5SCZVJiUWDv/BafMBwAADwhJREFUDA6RDm8Y6hrgnSLiYhqle1T1b0XkzcBJVb033u5O4D269tH2ucAfikgUf/bXVLXnjUWS3OzwVIlaM8API/JbyIHTa5YbPmGoXD8+gpODyUoh9dvWtdNlVpom9mouYHujtKzl2pkKK81gjS3qlTAueEZh1/LNbG4/iPByZgJmoeuaOjBWoN7O0WqHlFKw8+019VZAI86iWy3lUcz1OlstUC7kBurxItR0b+iyV8IPJ06c0JMnT27rM5FCO9x8u7SJkt8g1khv7hlgsQwGTy82WW4ECDBT9RiNb6gamYcQGG5VX5JavVu5paqdVOuDdGwCeO72Q1Eicr+qnthsu32f7uPCUp1qqUAxwzwuScVTq4KyDAFrssu2ArMOKMU2uQKoM1g30p3SOR66jmdAj2snDcV22F/D+eu479RlXv/Wk/z63z/CheVm5oO8e+HisuxdIjVGWbVmQBAayevUSIHEt+jScgs/Xr/X6vIwHE8rTHfMYl83FstNI997+Nwy9z0xz5m5BitNP/NGw2IZRIJQafkm5YUfmhT61XKeI5NlDk+WyDlC208/tb4lG/Z1GOolN03xxm99NqOFPNVinlYQGYmgxWJ5Bl5O8HI5SvGcnORpO58T8jgU43kHw/AU3ju60/Gkk5pnq9gwVIq44vDS5xzgQLVoFEGYpyeLxXIlVgd1uxuEJJcZ0JUteW+TzOZWNRLg7uWsjj8c5hncg06imz48WYq71koxQ9vHxG3MccQa2FuGhqYf0myHXK75jJZyVEv5LdmwDjqqih8qgpLPrRXA1NshzZaZ4T5ayuE6wkLdZ3q0QLWUzW01bensvm4sVsUN0vcUH+tp+SFn5hsA5Bzh6HS5o5KyWAaZYt6lkHNp+hGL9YClRsD0qEe1lB/aBkNVOX2p3ok0HJ0u4XU1GGXPpZR3TebmWC41XvEyFUoNbdbZYUAwU+QHYUyu5UcQ93TyufR7FXtRsWLJhiRTciu+kFSNTfAw169IwY8bCoFnjGUmx9y5jiT7a8qOWaRIpIPRUISRInFeFyfOv5FWKoQwMqGupYbfkT9aLL1gdqxAIe/gOkLdD4e6bjliDLxyjpBzheYGs3e7G4esG8e0pbP7umeRcLne5osXlqi3Q26aGeXweKkvP3wUKZdrbS7XfAQYr+SZGPFi06N0vj9SJYiUIDQ5b1w7U9yyCxrtkMsrbYJImazkOTLZn2snbUSMkVGWZkaDxr5uLBwxYaiTp+ep++bJYazUPze3lWbAQs0HoFLMxQ1Fuhda3nXIu1DMD/8ApCV7zi80O4kCt5PC3NJ7bBgqZYIIRgs5XBFcR1iO0xf0g8SaUcTk/+/nrds2FJZeYFRPpg63gwGI6e5jAiudTY9EOvv1x6e4uNxipRUwO1Lo2420XMhxfLZCvRVQGPIBQcv+5JrxAstNk2W26NneapaknfF9XzcWsYiBcwtN6q0QxMRgR4r9S9ntOsJoH0NfFksvObfQohmHcEueQ7lg091nhZPyad/fjQWQc4zkLzEzsXFXi2XrtIOoE/rwcrZ3nCW5YXXKGwYS6eyB8SIlzyXvSucpyWKxbM7BddfOMMtlh522lc6mT8lzOTxZyroYFsvQsZevnYVam8W6j4gwO5at580gsK8bC0eM3GwQ3PIsFsvgEEbKpeU2AIJS6ENWhd1ipbMpotiGwmKxPBNHjLRdBJDhyEZtpbMpYsOrFsv2UVUafghqkgg6actwMkBEuHa6TNOPzOTdIehZWOlsijjSSRhpsVi2yJNzjY5T3qGJ0p7NBiAiHaOnYcBN+ScY/OYyZYbggcFiGShMAkrTM8913aFUszP+sYA7rNJZESmKyKdF5HMi8qCI/NIVtvkBEbkoIp+N/364673vF5HH4r/vT6OMg5J11mIZJg5NlKgUXIp5Bz+ICEOl0Q6ZW27TaFv5bFYMs3S2Bdymqisikgc+LiJ/r6qfXLfdn6vqj3evEJFJ4E3ACUyU6H4RuVdVL6dYXovFsgWKnss13qpctt4KOHe5iWJc8/aqlHa/k1pjoebxYiV+mY//ttrufQvwYVWdBxCRDwOvAN7dyzIm3bYwWp0qH+nqWEa4g2UR011LBpscWbeMaf3ddcsaf2aj5fXl28ryRse2m+WdnIthOy+uk/hK76/zstnyRudlrJSjnC+z2PApey6e29/zstm5yOq89Lu+pB2GSnWAW0Rc4H7gBuB/qeqnrrDZd4rIS4BHgbtV9UngMPBk1zZn4nXr938XcBfAsWPHdlRGR8DpGsPqHs5y+ri8/rt7tbzRse1qn31c3kn5enJeZH+el02XNzgvrudQ9ApX/OyePRcMXn1Jk1S/R1VDVb0VOAK8UERuWbfJ3wDHVfWrgQ8D79zm/t+iqidU9cTMzExvCm2xWCyWZ9CXRklVF4CPYkJJ3evnVLUVv3wb8LXx8lngaNemR+J1FovFYsmANNVQMyIyHi+XgJcDj6zb5pqul68BHo6XPwjcLiITIjIB3B6vs1gsFksGpDlmcQ3wznjcwgHuUdW/FZE3AydV9V7gJ0XkNUAAzAM/AKCq8yLyy8B98b7enAx2WywWi6X/yF7RRJ84cUJPnjyZdTEsFotlqBCR+1X1xGbb2fnLFovFYtkU21hYLBaLZVP2TBhKRC4CX9nFLqaBSz0qTi8Z1HLB4JbNlmv7DGrZBrVcMLhl2265rlXVTece7JnGYreIyMmtxO36zaCWCwa3bLZc22dQyzao5YLBLVta5bJhKIvFYrFsim0sLBaLxbIptrFY5S1ZF2ADBrVcMLhls+XaPoNatkEtFwxu2VIplx2zsFgsFsum2J6FxWKxWDbFNhYWi8Vi2ZR931iIyCtE5Isi8riI/FzW5UkQkT8SkQsi8oWsy9KNiBwVkY+KyEOxXe5PZV2mhK1Y+WaJiLgi8q8i8rdZlyVBRE6JyAOxrfFA5csRkXERea+IPCIiD4vINwxAmZ7dZQP9WRFZEpGfzrpcCSJyd1z3vyAi7xaRYs/2vZ/HLOIkh49iMuKewSQufJ2qPpRpwYDYEGoFeJeqrvcByYw4U/A1qvoZERnFmFt9+4CcMwEq3Va+wE9dwco3E0TkP2Osgquq+uqsywOmsQBOqOrATS4TkXcC/6KqbxMRDyjHdgcDQXz/OAt8varuZkJwr8pzGFPnn6eqDRG5B/iAqv5xL/a/33sWLwQeV9UnVLUNvAe4I+MyAaCq/4zJxDtQqOo5Vf1MvLyMSSv/DBfDLFDDTq18U0VEjgCvwvi2WDZBRMaAlwBvB1DV9iA1FDEvA740CA1FFzmgJCI5oAw81asd7/fGYkv2rZYrIyLHgecDV7LLzYQ41PNZ4ALGx31QyvY7wBsw9tGDhAIfEpH7Y5viQeE64CLwjjh09zYRqWRdqHXcCbw760IkqOpZ4L8Dp4FzwKKqfqhX+9/vjYVlh4jICPCXwE+r6lLW5UnYgpVv3xGRVwMXVPX+rMtyBV6sqi8AXgn8WBz+HARywAuA/62qzwdqwCCNKXoYw7a/yLosCbFR3B2YhvYQUBGR7+3V/vd7Y2HtW3dAPB7wl8Cfqur7si7PldjIyjcjXgS8Jh4feA9wm4j8SbZFMsRPo6jqBeD9mNDsIHAGONPVM3wvpvEYFF4JfEZVn866IF38O+DLqnpRVX3gfcA39mrn+72xuA+4UUSui58U7gTuzbhMA008iPx24GFV/a2sy9PNVqx8s0BVf15Vj6jqcUwd+4iq9uyJb6eISCUWKRCHeG4HBkJ9p6rngSdF5NnxqpcBmYsoungdAxSCijkN/BsRKcfX6ctYtareNWnaqg48qhqIyI9j/L1d4I9U9cGMiwWAiLwb+GZgWkTOAG9S1bdnWyrAPCX/B+CBeGwA4L+o6gcyLFPCFa18My7TIHMAeL+5r5AD/kxV/1+2RVrDTwB/Gj/IPQH8YMblAToN68uBH8m6LN2o6qdE5L3AZzBW1f9KD1N/7GvprMVisVi2xn4PQ1ksFotlC9jGwmKxWCybYhsLi8VisWyKbSwsFovFsim2sbBYLBbLptjGwrJnEZEwzgz6YJyJ9mdE5Kp1XkSOi8jrd/BdR0Tkr0XkMRF5QkR+X0QKOy/9mn3/sYh8Vy/2ZbHsFNtYWPYyDVW9VVVvxujiXwm8aZPPHAe21VjEE6DeB/yVqt4I3AiUgN/YdoktlgHFNhaWfUGczuIu4MfFcFxE/kVEPhP/JWkRfg34prhHcvdVtuvmNqCpqu+IvysE7ga+L86h1UFEniMin+56fVxEHoiXf1FE7ou9CN4SN0Ks+/wpEZmOl0+IyMfi5YoYD5RPx4n37ojX3xyv+6yIfF5EbtzVibTsW2xjYdk3qOoTmJn6s5istC+Pk+i9Fvi9eLOfw3go3Kqqv32V7bq5GePr0f1dS8Ap4IZ16x8BPBG5Ll71WuDP4+XfV9Wvi/1LSsB2PC/eiEkj8kLgpcBvxjONfxT43Ti54glMziWLZdvs63Qfln1NHvh9EbkVCIGbdrnddrgH00j8Wvz/a+P1LxWRN2B8CCaBB4G/2eI+b8ckK/zZ+HUROAZ8Anhj7KfxPlV9rAflt+xDbGNh2TeIyPWYG/4FzNjF08DXYHrYzQ0+dvcWtnsIWDMALSJV4CDwRRF5B8b34ylV/VZMT+IvROR9GM+mx8TYX/4BxrXuSRH5b5gb/noCViMC3e8L8J2q+sV12z8sIp/CGC99QER+RFU/ssGxWiwbYsNQln2BiMwA/wcT6lFgDDinqhEmMaIbb7oMjHZ9dKPtuvlHoCwi3xd/lwv8j/i7Gqr6g3FY61sBVPVLmEbrF1gNQSU3/kvxOMdG6qdTwNfGy9/Ztf6DwE8k4xwi8vz4/+uBJ1T194C/Br56g/1aLFfFNhaWvUwpkc4C/wB8CPil+L0/AL5fRD4HPAdjrgPweSCMpbZ3X2W7DnHj8x3Ad4nIY8AcEKnqr1ylbH8OfC8mJJX4b7wVkyL8g5j0+Vfil4DfFZGTmAYn4ZcxIbPPx8f7y/H67wG+EGcIvgV411XKZLFsiM06a7H0mFgx9W7gOxK/cotl2LGNhcVisVg2xYahLBaLxbIptrGwWCwWy6bYxsJisVgsm2IbC4vFYrFsim0sLBaLxbIptrGwWCwWy6b8f1yeaeBR+X/7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch = eval_sampler.sample_n_episodes(actor=actor,\n",
    "                                       n_episodes=10,\n",
    "                                       max_path_length=None, \n",
    "                                       train_sampling=False)\n",
    "\n",
    "print(*batch.create_logs().items(), sep='\\n')\n",
    "\n",
    "batch.add_disc_cumsum(cumsum_field='q_vals', field='rews', gamma=0.99)\n",
    "data_q_values = batch.unpack(['q_vals'])\n",
    "\n",
    "model_q_values = critic.estimate_qvalue(batch)\n",
    "\n",
    "plt.hexbin(data_q_values, model_q_values, bins=50, cmap=plt.cm.Blues)\n",
    "plt.title('Data Q-values vs Critic Q-values')\n",
    "plt.xlabel('Data Q-values')\n",
    "plt.ylabel('Critic Q-values')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a4ea1b7",
   "metadata": {},
   "source": [
    "# Visualize trained actor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "3cccc38c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Simulation done, total reward: 6.0\n",
      "Loading video...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" autoplay \n",
       "                    loop controls style=\"height: 400px;\">\n",
       "                    <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFixtZGF0AAACrgYF//+q3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE1NSByMjkxNyAwYTg0ZDk4IC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxOCAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD0tMiB0aHJlYWRzPTcgbG9va2FoZWFkX3RocmVhZHM9MSBzbGljZWRfdGhyZWFkcz0wIG5yPTAgZGVjaW1hdGU9MSBpbnRlcmxhY2VkPTAgYmx1cmF5X2NvbXBhdD0wIGNvbnN0cmFpbmVkX2ludHJhPTAgYmZyYW1lcz0zIGJfcHlyYW1pZD0yIGJfYWRhcHQ9MSBiX2JpYXM9MCBkaXJlY3Q9MSB3ZWlnaHRiPTEgb3Blbl9nb3A9MCB3ZWlnaHRwPTIga2V5aW50PTI1MCBrZXlpbnRfbWluPTI1IHNjZW5lY3V0PTQwIGludHJhX3JlZnJlc2g9MCByY19sb29rYWhlYWQ9NDAgcmM9Y3JmIG1idHJlZT0xIGNyZj0yMy4wIHFjb21wPTAuNjAgcXBtaW49MCBxcG1heD02OSBxcHN0ZXA9NCBpcF9yYXRpbz0xLjQwIGFxPTE6MS4wMACAAAAB8mWIhAA3//728P4FNlYEUGa7Q91nCgDAQZ/NTMClgclA4pYaytdZh+dVJuV432kRnWAs5rNIwVgDq82rBm7oYytwVrbMx0s0deIjDqA1k01uG4ANrMfO1grOKrx25okAn/0B9ykWn5jDkOZjE0iaGxgeFI7/oNIxff+LCNKjIvQAVD7QlfIuXWn1BeVCqanKe2pch8+fSPvSKKs4uWmxHUbq8pOOGyf2r2qfhGMpi+pbq1wk2Csaw+nb81Loctum2EoJ95orDsIIp5EgIYmf3wQilWvDraoPNU5qoqIwLW7hN9tgAv2HsqNoAz5pHCrgIkzuE2d3Fk1bK9k/zF462NVORFSzsMs7cAADDbGFX1SuxUphNhf2FX6WVpEyw2AIwdulgy1LNDsaJidbZYoxK5fVaXJspSvhrvDxv6jSq3WW7sVUJox89h/GNZIGt4xyDrOw1+Sbb5spDSVc/4I06QDwIChjIWXp6/ZXUtzSN/PMr0Uy8/6g+QseFYbzGdJqy/w7+wk4fQdu5cWg8bGwMEXXf2s2VtPaJLfn3EWaf4vSpYNkbK2eOupC8DjGzWU5i/V8gaA1oV6VGQ8e3mJMd3FGersXylhMY7Hsq2orlNETctI7GUJgLwRh8JUgbHloZI5dGSc+nvkryzB+V3cb+CVeQQAAAEZBmiRsQz/+nhALmCWa6IjVS7Sb9uSf09GOZeXl3EdrMAJ1pcLQZbkTkb7WBLqwv2oyK1xYxP8+eAwFKREmoIMFged+jzW4AAAALEGeQniFfwGxOs2+oncKGbIVOC3y4AGzdi9kcwdShlOhGUyxAuT81ad5v7TBAAAAKQGeYXRCfwG38fMSSZdwBCG+ypUjG42HOox8HTqyNk/ff8rM1vgKNGNfAAAAHAGeY2pCfwB/ApQIzZkAIpqPpobMUpmgLNIGmlEAAAAzQZplSahBaJlMCG///qeEAcFaXgBQwdQLniLo3g8oxIA/9u3i/rgKWi7UCV/qt33adE/dAAAAdUGaiEnhClJlMCGf/p4QCs+7oz7U6F+uwAmXCtl92VAiwgKmPg6JU89l4AdsWRcJzqwwC230brEYN5uOqJGVTdudWdQCC4Z6AcB/eG7e/pgLnoT94H8vMbvOdWS5CwqHoxbOUQ6+3XnZGQmhhb0U4bvg7at+sQAAAEBBnqZFNEwr/wC6V7mZyjaEARMdV9d7FvPj0ERzeZeFGjSo3GzH+ATVgn4nOqXARDTLNe1+Enlbmev/kOlrczmBAAAARAGex2pCfwDtvC4oAIR5b9YmLb2rr3PuTMu1e59AhJYSRRuAF3t89ehOpNj2vFLKJVSIu3RcdWY4QeVmKl3ankBeCkGgAAAAcUGayUmoQWiZTAhv//6nhAEsOGAQATtwTPepTtFXbokCdBzusAuYtnR20+Pg5FBfBI2WbBT7TDtZIjhloDZAHCVi3xGkPEHpzl7ujeSnc4qVCGHYdlWEdBlhh04V05OHx5LukoHHo+EEJkI5H2SiNIewAAAAeUGa7UnhClJlMCG//hkX6RGrBZfjx3G7oCTIzAxMvAIOdj+2+N4k3Zr3esSX5bdqvpAgrgxPSVXJyNMeORLByZaac26h1mzpRt33ZPuEcWgNhRDZEUL65DlGn+fVIt6j+eUq1uqD3ZDRyDTAVZZn9q4NPvRpQVUl10EAAAAvQZ8LRTRMK/9XpcD9o6zIAH7KpMx6h9/o7w6cFK76CedrXEn+GahvN7YNzXGZWUAAAAAvAZ8qdEJ/AioZ8a3cEQAiryB0hq97l+REVwv0sZG1quwxGNIuU5oNU+KPLHEK58AAAAArAZ8sakJ/XxfD9lIAX1GMAEVeQOG+jvWHyIkIRNVitStMtviPmR9+BN+ngQAAAHZBmy9JqEFomUwU8N/+p4QBLfkbM6T2BwAnWr39PcIbKw+7Eavr7sJLsl+mO1lDcZwt7hllVsfqz1EAwztZIvpnbxDVRFt1szg0XgLnpYIWlx8eZR+hidj188Fj7zUgFCwLmZdZSgKzh50N/OP4cWmRVzFM1BnlAAAANQGfTmpCfwE9idRNAFb7PsqVIxr+apY5wWWoH3Oox8HMmT2kE4clyS+xG98E55c+HSOYDm+pAAAAUUGbU0nhClJlMCGf/p4QA0vseNEU4R29/NAYRc+IATjxqnkRfi0TqkbQw80LKa8OpTE1+LDrS0HTtoYexMI4l7xqHMYqsq++kzJ6fGN0YCS5TwAAADJBn3FFNEwr/wCxNVTy/26DFw0Nd4AHFHsXsjmDqUMp0Iyqp8NUkGmLWFqNTo7XcDoE3QAAADgBn5B0Qn8A4m02smfDcaPIRkAIq+PEMwPJh1mRtjFDGl6gNohWVQl+2AN2X8FqzKLbFTjpRXLwoQAAACQBn5JqQn8AfwKUBM256IAJ1jG5pRYDJou+1K93LDAQpG84oMAAAAAwQZuUSahBaJlMCG///qeEAHaLm3wAQkbqZqpYWzt4pMozEZwniaLNqoBURah79tmAAAAAb0GbtUnhClJlMCG//qeEAJ6cjfwAbH0vhjjgZhPG71Ctwyy6b2Z0vc77l3Uh5jUAsD/qsOrhE9zREdavqHiLbz3JoP5jXUC+gSqO5iUy8hvstYyStd7hOijQVgr/9Dfzj+HFmis2Lqe22s/BZvhLgQAAAL9Bm9lJ4Q6JlMCG//5CV5KSeG0sALFSn/+DzuBCh/KsSxOvjev3jThWJvocFh6fklsE+Tvg043ISP7ucQDZrwVpzOpA6m37LVHuX+zDX9mSFfSBSLsELuoSIp0HHFWEgqGWIFpCOV9txLKmwsWm8nMtqYSZ1dg/7izKZ2kiHPxEGiFyAE24amwV/jElrEr4q3au6m2rHBrtif/ThuvLK3Xp6RVK78NgEToyIk+HkpWzA3oLnFSSJO4ENVbUBzxplwAAAC5Bn/dFETwr/1elwPa0Vsa5eIAONrv7y8brNrYPje6I+I2zmSGRGDGBRMenoBGtAAAARwGeFnRCfwF80+Y9j4AR2aaC1GMl0V1Sxz+B6Coc3shspn0+LSK8UfvWVJsRjsV7QWAqg8oiae2HS8OfrU/vRlilGJKWNw8VAAAAKwGeGGpCf1/04e1sLnAA/nkrO9rXdkOMRPZiFdAA1+AU/BZs+GKaLswFIMAAAAArQZodSahBaJlMCG///qeEARxQfbN5UaACH0euCmwhF2XUHAzFHp84iD95QQAAAC1BnjtFESwr/0cgcDt4jw9ehbra2QBEwrF6wwT2QrOonts7eEB7vbawmX9AhEAAAAApAZ5adEJ/TC4puxY2wmtLgAi+GpYocIPbM0HXhhV0nzb3QMr2OmiOgiEAAAArAZ5cakJ/TRBpuy6L20WhAGfa71ACJjnA4b6O9YekynOFmbJmBsZw2GQKfQAAAC9BmkFJqEFsmUwIb//+p4QAp3T2+ACEpYWVMbTVxHX635Timy54Wp002+ZJNwrAgAAAADtBnn9FFSwr/0cgcDt4kD3hWCteUwV4HGezADgzETZ+CQGCHZB+PKX3WRIicdeSHbSfLFIQE0CXIGP1dAAAAC0Bnp50Qn9MLim7Fk7zmc5Jx67+h+EQAhzfaHoEkrbDm9kNlQVTM8IhJYIrH+MAAAAvAZ6AakJ/TRBpuy6L20PIMAC/2bUgvNJ8dCo6fpYg3PiO6CxJZDBXdWEZ1GaBJp4AAAAyQZqFSahBbJlMCG///qeEAevzVwAWqOeUzTAwhgvFVsQA1z6GvxhAnyIGE55w7T3fRoEAAAA4QZ6jRRUsK/9HIHA9HQMMpX0nzXG/6mv8ALRXJWdFVRWRcdJjNZ8bmRgwC9NA32AXKFCs3Q1N3TAAAAAnAZ7CdEJ/TC4puxZWW/A6kwARkKYepBV98+AwEYVcqZh9Q4tWcoCxAAAAKQGexGpCf00QacU02cID5OADgrNBajGTDpi5fkeD1K1JsHeInvtvDYaBAAAAX0GayUmoQWyZTAhv//4Fpe4Ah88f84KM3LrguvwMtIo51PNjKaz12nnCn3c4gFtQjyyZ2LtvsHxw4wOi+cGWAAwqHWeNHDiT2dbB11CRwWC9dxLfKGVXirJKZcdz2+q/AAAAL0Ge50UVLCv/Vv1N2j8lOaGQZuNIkgBtqq/QbCZK1hgs8W29biBc2B1lVBcK7KSBAAAAJwGfBnRCf151T9Ys0oUbizwBEhSqxqN96xBreznCzOVYAKhosOW5YwAAAC8BnwhqQn9NEGm7Lqnz/X84AR7yVVeIcfeZKv064hwijRER3vYclHWVnHHlpBjRoAAAAC5Bmw1JqEFsmUwIb//+p4QA3PsqQx4Kw/PeMAFqjnlGIvYubNfNJK0ZblEhz4ddAAAALkGfK0UVLCv/RyBwO3iSIhQmZSU1ACwD2L413X0SoXqLOVo9cY7LpYBMSMATx5IAAAArAZ9KdEJ/TC4puxZ1A3S83kfsEAIwKVWNRvvWINb2c4WZyru2JV1v9c10cAAAADUBn0xqQn9NEGm7Lqnzuo/Dl48AVqPgQkvunn6MaxKSzy8ZzhlizCc3sJj2SqGhcP+GesNS/QAAAFRBm1FJqEFsmUwIb//+BLQADjZ+h+jKNimYT8AU5Z7kmqgA7BRRAYBoNEdwVQDfFM0SaKqrsq8PbSTp0J+/IqQSRY1LcVpoWClFjz58T6J5BNFCg4EAAAAtQZ9vRRUsK/9XpabQi0pkZ/RPq3Jbf4qAB2grFqfPx8Zsb6UCb2pcP1aD7TeBAAAAIQGfjnRCf0wuKbsWdQOJVGRij/yaAEBePcGdA309zgYU9AAAABcBn5BqQn9f9OH4QzHUOTLrM63BVxDdQAAAAH9Bm5VJqEFsmUwIb//+p4QBrooaai/G27ZRTfa8AH5rQtCBOj+vbq/f5rOrrOKoiHJpnrFoqJLHxYotH1XXtqB/hGQpYdY9kGfUSArUFUsgyKhN9g8kx4uSMtBBOtfyHrQK9SCWpQSTRATaia0vw89n+JAVTPkEG06d2CnDvopTAAAALUGfs0UVLCv/RyBwPEIRuCHwephr/lEAHA8/vLxwMejWXyK3UW1+f01/6/9/WAAAACABn9J0Qn9MLim/Nnv6K90Lx4AiK7//v8uZEbU9nAUyOgAAACgBn9RqQn9NEGm7LqnzyuzjUGvABwIJ+KUcDHypVS0jVSSF2nVYioidAAAAaEGb2UmoQWyZTAhv//4FioABGEL//BwcBvkvQ1WZYttkslcEQbxi5LROHAZ7sYQEfuFwAoEfcWCpCDgUV20F5HyJ5pKtD5u6SLUhGXhP1zoUQBTjpIDnf5/JvOlQHblRR1onTSb5lC/mAAAAL0Gf90UVLCv/V6Wm0ItKZGd/gHD8+tf/ryAGrvfRQ2Y8PRrL5VUV53TeAe0Wuc2BAAAAOwGeFnRCf0wuKbsWdQR475gas0AQcN/FKOCWrj1xIyhUL2k9ZOJ0koXE1U2xGCB9Gamovy75gOYN8ceLAAAAIwGeGGpCf1/04fhDQtngA4EE/FKOBj5UqpaRq08ZwXSllyNgAAAAOkGaHUmoQWyZTAhv//6nhALeT9S1rVwBE1yXbdYknKBF6GtujZXCUAOgt/ikbWvt3Cixfr5/rVVA+YEAAAAnQZ47RRUsK/9HIHA/YxPZIs5A47Ef/nAAHVSuA2D5jAk5RfnzSUuoAAAAIQGeWnRCfwIY1RiM4AOLbvvzxwBhW6NikUKTk545FZ/wkQAAAB0BnlxqQn8BpHm74Y6o+gyieNACAzQoh2iZB6pjsQAAAChBmkFJqEFsmUwIZ//+nhAGN09LAB+A/+8KOBm8M0oAvYxZE06lhKveAAAAL0Gef0UVLCv/RyBwPEdr17EuRM1FkbwAbbw44NRjcbHPeR+Dp1VrWJlUulP3ejNAAAAANAGennRCfwGj9Hm8l4Bg6hiT2iIAWCPgQkvunn6MaxKSzy8ZzhlVhSQl/77oApro3A8T/ekAAAAfAZ6AakJ/AaR4C1j0YA3fVrwBEtCf/3+Tq4AexvCf8AAAAI5BmoVJqEFsmUwIX//9bJ4svleJmcAFwo+Ffkf8xtl52RhuoCvy/aYJKV3wMIL64633DJLy3Gmkx/YNomnmzpZHe4ZT06LP2ihe7amHRI3PmJcn4I5p2i9ddzmDJvghoZLSmCjIfob1KpDG+vU3MzLOmHKSdrF7AiGGoASne4LhuyNum1/5GE6mHXF4Tc2fAAAALUGeo0UVLCv/Vv2B9TlmSgfGkFbjz/cUALVnH/7/LmLNZx7xF3iskh+KNlwXkAAAACMBnsJ0Qn9edU/qJRUO5pwAOLhe8uCqoFwGAT/Z9IGWObjcgQAAADQBnsRqQn8BpHm74Y6ooOb7zwBWoxuaVIxsed7yyIsFMOdRj4OZ6t1XmOcc/iY1gF6VV5CBAAAAYEGaxkmoQWyZTAhX//44QAafVXAzJkwAlkqjrbw1sqR2Y0QFkeGi8CFwYY48tvDGk15s74BtSVI0KGOMhlcj8dAqoIgvC1TuTqvAKn2W/idZuD38GQC+yxwEgkWbG8Np3QAAAFlBmudJ4QpSZTAhP/3XtSgAnTP/8YujXOtLZHD7O+LazyWfEQf6KaBCAObR+CRBKy2RtRCzZ4lbXz+oyf9ZHeMXp2B7YFbT6zJOFDWf8BVKlvC9rXw8svtCSQAABlttb292AAAAbG12aGQAAAAAAAAAAAAAAAAAAAPoAAAJYAABAAABAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAFhXRyYWsAAABcdGtoZAAAAAMAAAAAAAAAAAAAAAEAAAAAAAAJYAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAAAoAAAANIAAAAAACRlZHRzAAAAHGVsc3QAAAAAAAAAAQAACWAAAAQAAAEAAAAABP1tZGlhAAAAIG1kaGQAAAAAAAAAAAAAAAAAADwAAACQAFXEAAAAAAAtaGRscgAAAAAAAAAAdmlkZQAAAAAAAAAAAAAAAFZpZGVvSGFuZGxlcgAAAASobWluZgAAABR2bWhkAAAAAQAAAAAAAAAAAAAAJGRpbmYAAAAcZHJlZgAAAAAAAAABAAAADHVybCAAAAABAAAEaHN0YmwAAACYc3RzZAAAAAAAAAABAAAAiGF2YzEAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAoADSAEgAAABIAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAY//8AAAAyYXZjQwFkAAz/4QAZZ2QADKzZQod+IhAAAAMAEAAAAwPA8UKZYAEABmjr48siwAAAABhzdHRzAAAAAAAAAAEAAABIAAACAAAAABRzdHNzAAAAAAAAAAEAAAABAAACOGN0dHMAAAAAAAAARQAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAgAAAAAAgAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABgAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAgAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAACAAAEAAAAABxzdHNjAAAAAAAAAAEAAAABAAAASAAAAAEAAAE0c3RzegAAAAAAAAAAAAAASAAABKgAAABKAAAAMAAAAC0AAAAgAAAANwAAAHkAAABEAAAASAAAAHUAAAB9AAAAMwAAADMAAAAvAAAAegAAADkAAABVAAAANgAAADwAAAAoAAAANAAAAHMAAADDAAAAMgAAAEsAAAAvAAAALwAAADEAAAAtAAAALwAAADMAAAA/AAAAMQAAADMAAAA2AAAAPAAAACsAAAAtAAAAYwAAADMAAAArAAAAMwAAADIAAAAyAAAALwAAADkAAABYAAAAMQAAACUAAAAbAAAAgwAAADEAAAAkAAAALAAAAGwAAAAzAAAAPwAAACcAAAA+AAAAKwAAACUAAAAhAAAALAAAADMAAAA4AAAAIwAAAJIAAAAxAAAAJwAAADgAAABkAAAAXQAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1OC4yOS4xMDA=\" type=\"video/mp4\" />\n",
       "                 </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_actor(env=eval_env, actor=actor, nsteps=1000, train_sampling=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e32d683c",
   "metadata": {},
   "source": [
    "# Save Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "67872fd6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved checkpoints for ArgmaxQValue...\n",
      "exploration_global_step global_step\n",
      "Saved checkpoints for CategoricalDQN...\n",
      "models_state_dict global_step local_step n_updates n_target_updates optimizer scheduler\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/nikita/.local/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:200: UserWarning: Please also save or load the state of the optimzer when saving or loading the scheduler.\n",
      "  warnings.warn(SAVE_STATE_WARNING, UserWarning)\n"
     ]
    }
   ],
   "source": [
    "actor.save_checkpoint('trained_models', 'argmax_q_value_actor')\n",
    "critic.save_checkpoint('trained_models', 'categorical_dqn_critic')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31e0a904",
   "metadata": {},
   "source": [
    "# Load Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12804137",
   "metadata": {},
   "outputs": [],
   "source": [
    "actor.load_checkpoint('trained_models', 'argmax_q_value_actor')\n",
    "critic.load_checkpoint('trained_models', 'categorical_dqn_critic')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python Relax",
   "language": "python",
   "name": "relax"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}