2.sarsamax.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Temporal Difference Control Methods - Part 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In monte carlo control methods, we had to wait for the episode to finish in order to make Q table updates. In temporal difference (TD) methods, we update the table as soon as the agent starts interacting with the environment (i.e every timestep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Part 2**:  In this notebook, we will implement SARSAMAX (off-policy method) to estimate the optimal policy of CliffWalking gym environment. Sarsamax is also known as Q-Learning.\n",
    "\n",
    "\n",
    "--- \n",
    "So, What is the difference between on-policy and off-policy methods ?\n",
    "\n",
    "**On-policy** methods use same policy (<span style=\"color:blue\">eps-greedy</span>, that is used to evaluate and improve) to select agent's action.\n",
    "\n",
    "**Off-policy** methods use different (<span style=\"color:red\">greedy</span>) policy to select agent's action, and it is different from policy (<span style=\"color:blue\">eps-greedy</span>) that is evaluated and improved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import sys\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from collections import defaultdict, deque\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('ggplot')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gym Environment\n",
    "\n",
    "Create gym environment and explore state and action space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# simple environment with discrete state and action space\n",
    "env = gym.make('CliffWalking-v0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State space: Discrete(48)\n",
      "Action space: Discrete(4)\n"
     ]
    }
   ],
   "source": [
    "# Explore state and action space\n",
    "print('State space: {}'.format(env.observation_space))\n",
    "print('Action space: {}'.format(env.action_space))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us see how well does the random player performs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "last reward: -1, total score: -15071\n"
     ]
    }
   ],
   "source": [
    "# Random player: takes random action\n",
    "env.reset()\n",
    "score = 0\n",
    "while True:\n",
    "    action = np.random.randint(env.action_space.n)\n",
    "    state, reward, done, info = env.step(action)\n",
    "    score += reward\n",
    "    if done:\n",
    "        print('last reward: {}, total score: {}'.format(reward, score))\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We continue using epsilon-greedy policy (same as monte carlo implementation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def eps_greedy(eps, Q, state, nA):\n",
    "    rand = np.random.rand()    \n",
    "    if rand < eps:\n",
    "        return np.random.randint(nA)\n",
    "    else:\n",
    "        return np.argmax(Q[state])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SARSAMAX\n",
    "\n",
    "Also known as Q-Learning\n",
    "\n",
    "Update rule ->\n",
    "$Q(S_t, A_t) = Q(S_t, A_t) + \\alpha(R_t+1 + \\gamma \\, max_a \\, Q(S_{t+1},a) - Q(S_t, A_t)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_sarsa_Q(alpha, gamma, Q, state, action, next_state=None):\n",
    "    next_action = np.argmax(Q[next_state])\n",
    "    Qsa = Q[next_state][next_action] if next_state is not None else 0\n",
    "    updated_q_value = Q[state][action] + alpha * (reward + gamma * Qsa - Q[state][action])\n",
    "    return updated_q_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_sarsa_episode(env, Q, eps, alpha, gamma):    \n",
    "    nA = env.action_space.n\n",
    "    state = env.reset()\n",
    "    score = 0\n",
    "    while True:\n",
    "        action = eps_greedy(eps, Q, state, nA)\n",
    "        next_state, reward, done, info = env.step(action)\n",
    "        score += reward\n",
    "        \n",
    "        if not done:\n",
    "            Q[state][action] = update_sarsa_Q(alpha, gamma, Q, state, action, next_state)\n",
    "            state = next_state\n",
    "            \n",
    "        if done:\n",
    "            Q[state][action] = update_sarsa_Q(alpha, gamma, Q, state, action)            \n",
    "            break\n",
    "            \n",
    "    return Q, score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Play for defined number of episodes\n",
    "def train(env, num_episodes, eps=1.0, eps_min=0.01, eps_decay=0.9, alpha=0.01, gamma=1.0, plot_every=100):    \n",
    "    Q = defaultdict(lambda: np.zeros(env.action_space.n))\n",
    "    print_every = int(0.1 * num_episodes)\n",
    "    print_every = print_every if print_every > 0 else 1\n",
    "    all_rewards = []\n",
    "    tmp_rewards = deque(maxlen=plot_every)\n",
    "    avg_rewards = deque(maxlen=num_episodes)\n",
    "    for i in range(1, num_episodes + 1):\n",
    "        eps = max(eps * eps_decay, eps_min)        \n",
    "        Q, score = generate_sarsa_episode(env, Q, eps, alpha, gamma) \n",
    "        all_rewards.append(score)\n",
    "        tmp_rewards.append(score)\n",
    "        \n",
    "        if i % print_every == 0:\n",
    "            print('\\rProgress: {}/{}, average: {}'.format(i, num_episodes, np.mean(score)), end='')\n",
    "            sys.stdout.flush()        \n",
    "            \n",
    "        if i % plot_every == 0:\n",
    "            avg_rewards.append(np.mean(tmp_rewards))\n",
    "    return Q, all_rewards, avg_rewards"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the agent and try to recover optimal policy (or near optimal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Progress: 5000/5000, average: -13.0"
     ]
    }
   ],
   "source": [
    "# Hyperparameters (RL is very susceptible to hyperparams)\n",
    "\n",
    "eps = 1.0       # starting epsilon\n",
    "eps_min = 0.01  # minimum epsilon\n",
    "eps_decay = 0.9 # decay rate\n",
    "alpha = 0.01    # Q value update step size\n",
    "gamma = 1.0     # discount factor\n",
    "\n",
    "Q_sarsa, score, avg_score = train(env, 5000, eps=eps, eps_min=eps_min, eps_decay=eps_decay, alpha=alpha, gamma=gamma)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "policy = np.array([np.argmax(Q_sarsa[key]) if key in Q_sarsa else -1 for key in np.arange(12*4)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  0,  1,  1,  1,  1,  1,  0,  1,  2,  1,  2],\n",
       "       [ 0,  1,  1,  1,  2,  1,  1,  2,  1,  2,  1,  2],\n",
       "       [ 1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  2],\n",
       "       [ 0, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# UP: 0, RIGHT: 1, DOWN: 2, LEFT: 3, N/A: -1\n",
    "# Display policy\n",
    "policy.reshape(4, 12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot average rewards over the total number of episodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAEaCAYAAAAotpG7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3XecnFXZ//HP2d00AiFlQ0ihapBepCqIKF1FrBfFR0DxiQXsBRF9ROHhh/hYsIBGRKrgpYiAgEjHQkdQikqkhoTAkkbKzia79++PcyaZTGZ37y2zu7Pzfb9e85qZu811ljDXnHKfE7IsQ0REpFoaBjsAEREZ3pRoRESkqpRoRESkqpRoRESkqpRoRESkqpRoRESkqpRoRKRfhRA+FkJY1sNztg0hZCGEPaoVlwweJRoZECGEMSGEM0IIT4YQVoYQXgkh3B9C+FSFY0eHEBaGEJaHECZW2H9R+lLKQgjtIYS5IYRLQgjTy46bFEL4QQjh6RBCIYTwcgjhTyGEYypcc3o6Zl4IoamHZftJiuMTPTmvv4UQ9kt/ky27OOaAkr9dZ4+L+hjKxcDWPTznSWAq8HAfP1uGoB79DyXSB+cDbwE+DTwCjAN2AzavcKwBTwMLgOOB71U45k/puEbgNcCPgV8Dbyw55ipgPPBR4F9AM7A3MKnC9U4Efg9sCxwBXJ2nUCGEscAxwFnpc87Lc94g+ivxC73oC8B7gTeUbFtZ6cQQwsgsy9q6+4Asy1Z2do0uzmkHXuzJOVJDsizTQ4+qP4DFwMk5j/0z8EngKODxCvsvAm4p2/ZJIAPGpffj0/t35Pi8BuBZYoI5BbixB+X6CPAgMApYCOxd4ZhjgP8ArcQv+nek2PYrOea1xMS4GFgE/BHYqWT/CcBqYF/gIWBF+tw90/4t0zVLH3fkiP90YE6F7dumaxyVYlkBfAMYAfwceIqYTP5T3F5y7seAZeXvgQOIPzJWAPcBu1X4vD3K3r8HuDGdMwc4tizOmcCt6W/7LPDfwD3Ajwb737weax9qOpOBMh84rFJTWKkQwvbAnsAvgd8BU0MI+3dzzjTgfUB7ekD8YnsVODLVOrpyODFR3AhcChzYVfNTmY8CF2VZVgB+ld6XxrY7cDlwBbALcA7w/bJjphCT60vAm4B9iDWwO0IIk0sObQD+H7FW+Pp0vKemvueBI9NxexFrLe/JWYaunANcCOwA/IJYg3wBOBrYjlgj+kR67sooYlL7OLA78b/NlSGE7r6DvgX8DNiZ+O/houJ/mxBCI3ANMBLYD3gXsZa7ff7iyYAY7EynR308iL/EnyUmgr8Ds4lfDKHsuHOBq0re/wS4rOyYi4i/7pcRf+kWf8H/X9lx7wZagDbggXTtt1aI7RrgOyXv/wCcmaNMuwEFYFJ6vw+wnFSrStsuB/5Udt7HKKnREL+A7yk7JhBrC59J709I57y+5Ji907bXpff7pfdb9uC/y+l0XaP5Yo5rnAr8o6x85TWaDNi+ZNub07Ytyj6vvEbziZJzRqa/9/Hp/RHp39PmJcdMSceoRjOEHqrRyIDIsuwvxL6UNxE7i6cAvwGuDSEEiIMAgA8SE0nRxcB7K9SE7gV2Jf56PwO4G/hq2WdeDUwHDiM2S20P3BpC+HHxmDSA4O0VPvPDOQYFfBT4fZZlr6TPuweYC/xXyTHbE5tySt1d9n5PYPcQwrLig/iLf0ti09CaIhGbnormpecp3cTZF/eVbwghfCIN5Hgpxfp1YIturlMAnih5nzf2NYMDstg/1FJyzvbAvCzLnis5ZgGxWU+GEA0GkAGTZdlqYh/FX4HvhBD+i9hUtT9wJ7HZYwJwdco9RY2sPyhgZZZlc9LrR0MIrwF+SGyjL/3MAnBbevy/EMJXgTNCCN/OsuwZ4iCARuBvFT6z00EBqTnuWGBsCGF1ya4G1h8U0N0U6Q3EfoaTK+xbUvK6I4ud5uXXreYPxuWlb0IIHwS+C3wJ+AuwlJhYT+nmOquzVOVI8sZePvggKztH08/XACUaGUzFX7ibpOdZxJrFd8qOO4GYQCqNPis6HXgihPDTLMseyPGZk0MIzxETzVnEPpRSX0nxdDb67Bhi892urPtlN5HYt7J3lmX3Ao+z7oguiE1spR4glvGFLI7Y6q3il3JjH67Rnf2Be7Ms+0FxQwhhqyp+XlceB6aHEDYv1mpCCJsQh1bfOkgxSQVKNDIgQgh3Er/MHwBeJo6yOos4yur2NAhgX+DULMseLTv3p8DnQwj7Z1l2V6XrZ1n2ZAjhOuB/gUNDCJOIzWW/IDY3LQZ2JHamP01skjkc2Az4aWnzS/rMi4AbQwhbpppPuY8CV2dZ9o8KZb0n7b+X+Ov//hDCN4HLiH0Pny+GnZ5/REx4vwshnEns2J+R4rs+y7K/VipzBc8CHcDbQgi/AgpZli3p5pye+hdwdAjh7en1u4ij6AbDDSmGS0MInyOW/WxiM51qOkOI+mhkoNwIfIC1Xw6/IN6kt2+WZS3EL+Z5xNFX68iy7EliYpjVzWd8GzgkhHAAcaDAX4GTiM1mTwA/SK/fnGXZqnS9e8uTTHIbcbjyR8p3hBB2BfYAvJM4fgUcFULYOMuyB1O5PwD8g9hxXuxLak3lW0Cs9bQAvyX+fS4n9nvM76bMa6TrnAp8OZ13Td5ze+CHxPuVLiMOr94ZOLMKn9Ot1Ix4JHFAwF+Io9J+AzxD+tvK0BDWbTYVkWoLIRxHTLSTsixbPNjxDCchhPHE4defybLsZ4Mdj0RqOhOpshDCF4DbiTWkPYn3hvxaSabvQgjvIQ5x/yfx3qFvEmszvx3MuGRdSjQi1bczsV9mIrH/5TLikGDpu7HEvr4tiEPC7yc2x74yqFHJOtR0JiIiVaXBACIiUlVqOotUrRMR6Z3Q3QFKNMm8efO6P6iC5uZmWlpa+jmaoU1lrg8q8/DX1/JOmzYt13FqOhMRkapSohERkapSohERkaoaln00ZnYYce2RRuACdz97kEMSEalbw65GY2aNxPXjDyeuV3GMmWnFPRGRQTLsEg1xIaw57v6Uu7cBV7J2iVsRERlgw7HpbDpxmo+iucQlb9dhZrNIswG7O83Nzb36sKampl6fW6tU5vqgMg9/A1Xe4ZhoKt08tN4Nme4+m7huPUDW27Hk9TbuHlTmetFdmbOVK2DJQmhohJGj0mMkNDZRtlppj2QdHdC+Oj5Wr4b29vScXpfuaytAW4GsrQCF1vi+UICGACNGxphGjCQU42toWOc62ZrrxecNR49h2ZLFJdvaOw909GgYM5awwYawwVgYs0F8bmgsibOT2FP8WXs7dHRAYyOhqQkaR0BjIzQ1xUdjU3xf/txWgOXLYMVyshXxmRXLYlxjN4QNNoxxpdeMHLXu37N9FbS3s/HYsSwZ3xz/Pr2Q9z6a4Zho5hIXsyqawdr1yUXqUtbRDqtWQVsbrGpLX8itUFgJhVay1vR6VVv84mtvZ/mYMXS8ujS+X9UGixeSLWqBRa/A4legtZPFQBsa1iaeUaNLklB6QPrs9GhrhdbW+BmrV0PW0f/lz3ncq1W89kBdpyfXWwQ0fPM8mDqjnz99XcMx0dwPzDSzrYjrUhxNXNtdpCZl7e2w/NX4i3XVqviFXHxe3UZWKMCrS2DpIliymGzpIliyCJYuWZs8uvpl3ollpW9CA2w8ASZMgmmbE3bYDSY0x21ZtqZmseZRaF2T0LJiTaOtAEsXx+NHj4Zx4+Mv6dGjYdSYWAMp/RW/3i/6pvTLf8TabWsSWlliyzpKkmobrEqf39HReS2hsYlJkyfzytKla7c1NEKl2lmWxQS5Yvnax8rlZMuXxc9O1wuNjWtrKY2N0DSirGxNsfbV3hFrGatXr60BldWE1ql9jRy1tia1wYYwdiyM2RDIUk0nPZanGs+qwpo4QvGzm5oYN3ESSyeq6azH3H21mZ0M3EQc3nyhuz82yGFJHViTEJa/CsteheVLyZa9uvZXekfH2udUa6AtJovS5JG1roRlS+M1li2NXxh5ZllvbIRxE2DceJjQTNjitTB6g9icNWJE/CIvPkaOIowaHb+YR4+BkaPjF/6IkfE6oSF+6S5eHJNMQwOhocbGDo3eoMenNIyfSFido0YVQrz+6A1g4uS1m3v8ifnlvvb4ifGR47xRzc2EAWgSHnaJBsDdbyAuGSzSr7KODnjlJZj/PNn8uTD/ufi8YF5MMD0Vin0JI6EpJYRRY2CjcYTNJ8OG4+Jjo3Gx3X1NshgRfx0XX280HsZu2K/JoGHMBoTlK/rtelK/hmWiEelK1tERk8LihbDwZbKWl+CVBWSvvAQtL8VEsmJ57GtIj5caG8kIqebRtvZiG0+AqZsR9twvftlvuBFsOI6w4UYwdlx83zQiXieEtdcMDWubgvrQcS5SC5RopKZlhdbYOb1kIdmSRalzua2sz6CVbOnimFgWL4z9F+2r173QyJEwaQpM2oSw9Tax3bukmWvMqFGsXL481iA2nU6YullMMGM3HJyCi9QQJRoZ8rL29thU9dx/4LmnyOY/n0Y+LYSVy7s+udhZvNHGMH4i4XU7xvbrjScRxk+AiZtA8yaxFtJFzWKj5mYKdTa8WaS/KNHIoMpWLI8d3ivXjt7JViyL71+aT/bcUzD3mbXNVaNGw9TNYOoMwrY7x1FQ4ycRxk+MzVijx6xNLiNGqllKZAjIlWjMbDNgF2A8sBh4xN2f7/oskdQfsqgF5s8la3kRWhaQvbwAWtJjxbLOTx4zFjbfmvCWt8HmryFs/hqYMpXQ0DhwBRCRPus00ZjZCOCj6bE1MId4P9NGwGvN7GngJ8DsNKeY1LGs0Aovz4cF82PT1vy5ZC8+Dy++EPtJippGxKaq5imErV8HzVPi/RQbjI2JZYOSx6gxqpGIDANd1WgeAW4jJpp73X3NHV9phuS9gA8AfwN2qGaQMjRkWQYtC2h98lE6/vVYbNp6+cWYYJYsWvfgiZNj89Y2O8KmMwhTZ8DkqbDxhNq7H0NE+qSrRHOAu79UaUdKOncDd5vZ5ErHSG3LOjpg3rNkTz8Jzz9NNvfp2FeycgVLigeNnwSbbErYcXeYvClsMpWwyVSYMp0weswgRi8iQ0mniaZSkjGzBmCKu88vOe7lKsUmAyjLMljwAtk//072z7/Dvx6NnfQQbyDcbEvCPgfAjK2YsNNuLN5gY8Ko3k3EJyL1Je9ggPHAecD7gFXAWDN7J7CXu3+1ivFJP8s6OuKw4JdfjJ3zL70IL80jm/N43A5x+pKd9oBtdyK8ZjuYvOk6zV0jBmjaChEZHvIOb/4JcaLPLYDH07a7ge8ASjRDXNbRQXbP7WQ3XxM751evWruzoQEmTibM3CEmlm13hslT1QkvIv0mb6I5EJjm7qvMLIPYZGZmm1QvNOkP2T//TsevL4TnnopDhN/6Dpg8hTB5auxXmTg5zuYqIlIleb9hlgDNwJq+GTPbvPS9DC3Zi3Pp+M1F8Mh9MZl85POEPd+kEV8iMuDyJpoLgKvM7DSgwczeAJxFbFKTISRbMI/s1uvI7vpDvDP+PccRDjyi1yvoiYj0Vd5E8y2gFfgxMAK4EPgpcG6V4pKcso52eOpfZA/fR/bIffDi3Lh2yP6HEo44hjBu/GCHKCJ1LleicfcM+H56yBCQPf1vsjtvJPv7A3F1xcZG2GZHwlveRthlb8Ik3d4kIkNDV1PQvDXPBdz9tv4LZ70Yvg0cAbQB/wE+5O6L075TgROBduBT7n5T2n4YsabVCFzg7mdXK77BkK1aRXbdL8n+cDWMHkPYaXfYdW/CDq+P07iIiAwxXdVofl72fjqQAa8Ak4grhM4lzoNWLTcDp6blmb8FnAqcYmbbA0cTp76ZBtxiZtukc34MHJxiu9/MrnX3xytcu+Zkzz9Nx4Xfg7nPEPY7mGAnEsb0fLlaEZGB1NXMAFsVX5vZV4jJ5WvuvsLMNgC+SUw6VePufyx5ew/xhlGAI4Er3b0APG1mc4hzrwHMcfenUtxXpmNrOtFk7e1kf7iK7LorYeyGNJz8NcIuew52WCIiueQd6/pZ4MvuvgIgPZ8KfK5agVXwYeDG9Ho6ULpMwdy0rbPtNStbMI+Oc75M9rvLCLvuTcPpP1KSEZGaknfU2XJijeEvJdv2BFb0NQAzuwXYtMKu09z9mnTMacBq4PK0r9Jt6xmVE2fWyefOAmYBuDvNzc09jDxqamrq9bndKTx8L0vOOY3Q2MhGnzud0fsdPCTu2K9mmYcqlbk+1FuZB6q8eRPN14A/mNl1xBrDZsA7gJP6GoC7H9TVfjM7Pn3WgWn0G8SaymYlh80A5qXXnW0v/9zZwOz0Nmvp5dxdzc3N9PbcrnT8+WayS38MUzej4ZP/w/JJk1n+SlVbKnOrVpmHMpW5PtRbmfta3mnTpuU6Lu/w5kvN7EHgvcTO938CZ1a7kz2NIDsFeHOx2S65FvilmX03xTMTuI9Y05lpZlsBLxAHDBxbzRj7W5ZlZL+7nOwGh+13o+Fjp6jDX0RqWu5Jrtz9cTP7JzAFWODuHdULa40fAaOAm80M4B53/5i7P2ZmTuzkXw2cVFyYzcxOBm4iDm++0N0fG4A4+0W2ahXZRT8gu+9OwpsOIRz7Mc1DJiI1L2RZxS6MdZjZOOKX/lHE5LQauJJ4/8qSrs6tEdm8eRVb2LrVX1XtbPmrdJx3Fvz7McK7P0g4/H1Doj+mknprXgCVuV7UW5n7qems2y+qvKPOfgCMBXYCNih5/kEv45MS2bKldJx9Cjz1L8JHPk/D294/ZJOMiEhP5W2XOQzYuqSf5N9m9iHi3frSB1l7Ox2zvw0tL9LwmW8QXrfTYIckItKv8tZoWoHyybOagUL/hlN/sqsvgSceif0xSjIiMgz1ZJmAm9Mor2eJK21+lrXDg6UXOu7/E9lNVxPefBgNbzpksMMREamKvInmf4n3oxxLHE48DziHuFyA9EI292myi34Ar9mWcPR/D3Y4IiJV05NlAi5EiaVfxBFm/w/GjKXhY18mNI0Y7JBERKomV6Ixs2OAh939iTRL8s+I0/N/wt3/Wc0Ah5uso52On/0fLGyh4YtnEcZPHOyQRESqKu9ggDOBhen1d4D7gbuA86oR1HCW/e5yeOxvhGNnEV6z7WCHIyJSdXn7aCa7+wIzGw3sR5yufxVQP3c29YNszhNkN/6GsP+hNOx/2GCHIyIyIPLWaF42s9cChwP3p3VgRpPjjlBZK7v3Thg5kmAnDnYoIiIDJm+N5gzgQWK/zFFp24HAI9UIajjKOtrJHvor7LQHYdTowQ5HRGTA5KrRuPtFwFRghrvfnDbfS5wdWfKY8wQsXUzYfd/BjkREZEB1WqMxs1Bc/8XMGoizAxRfg/pneiR78K8wYiRhpz0GOxQRkQHVVdPZEmBcer2a9VeqDGlbYxXiGlayjo7YbLbj6wmjxwx2OCIiA6qrRLNDyeutqh3IsPbUP2HxQjWbiUhd6jTRuPvzJa+fhdicRpxMs6VkWWXpRvbAX6BpBGHnPQc7FBGRAZd3ZoDxwA8BA0YAbWb2a+DT7r6wy5P7gZl9Afg28X6elpTwzgXeBqwATnD3h9KxxwNfTaee6e4XVzu+rsRms7thh920JLOI1KW899H8AhgD7ApsCOxGXGK56nOfmdlmwMHAcyWbDwdmpscs4Px07ETg68DewF7A181sQrVj7NLT/4ZFLYQ91GwmIvUp7300bwGmuvvK9P4JMzuBOItztX0P+BJwTcm2I4FLUvPdPWY23symAgcANxdrWWZ2M3HRtisGIM6Ksgf/Ao1NhJ33GqwQREQGVd4azb+ALcu2bZ62V42ZvRN4wd3LbwydDjxf8n5u2tbZ9kGRZVkc1rz9roQNxg5WGCIigypvjeZW4I9mdinxi3wz4L+AS83sw8WD3L3HTWlmdguwaYVdpwFfASqtCFZp6pusi+2VPncWsdkNd6e5uTlXvOWampo6PXfVk4+zcOHLjPvARxnTy+sPRV2VebhSmetDvZV5oMqbN9G8AZiTnt+Qtv0HeGN6QPxC73GicfeDKm03s52Iw6ofMTOAGcBDZrYXsaayWcnhM4jNeHOJzWel2+/o5HNns3aF0KylpXf3nzY3N9PZuR23Xg+NjSx7zfYs7+X1h6Kuyjxcqcz1od7K3NfyTps2LddxeRc+e0uvI+kld/8HsEnxvZk9A+yRRp1dC5xsZlcSO/6XuPt8M7sJOKtkAMAhwKkDHDpQ0my23S6EsRsORggiIkNC3hoNZjaJOJx4U3f/tplNAxrcfW7VouvcDSmWOcThzR8CcPeFZnYGcb0cgG8OxPDrip57CloWEN5ug/LxIiJDRd77aN4MXAU8AOxLvKdlJvAF4IiqRVfC3bcseZ0BJ3Vy3JBYcjp78M/Q0EDYde/BDkVEZFDlHXX2feAodz+MOO8ZxNmbNWa3gjXNZtvuTNhwXPcniIgMY3kTzZbufmt6XRzF1UYPmt7qytxn4KX5mttMRIT8ieZxMzu0bNtBwD/6OZ5hIXv0QQDCbvsMciQiIoMvb43k88Dvzex6YIyZ/ZTYN3Nk1SKrZctejWvPbLTxYEciIjLo8q6weQ+wM/AYsaP9aWAvd7+/yxPrVVsrjBo12FGIiAwJuftY3H0ecE4VYxk+Cq0wSguciYhA/j4a6YGsUICRqtGIiIASTXUUVsKo0YMdhYjIkKBEUw2FghKNiEjSkyloJgAbAa+6+6LqhTQMtLXC+EmDHYWIyJDQZaIxsxHAN4hziW1CnIY/M7MFxFU3T3f3VVWPstYUCgTVaEREgO6bzs4nLgvwAWKiGQlMJq5Fs0/aL+UKKzUYQEQk6a7p7L3E6WeWlGxbCNxmZn8j3k/zkWoFV7PURyMiskZ3NZpWYGon+zZN+6VcW6sSjYhI0l2N5hzgdjP7OfAIsAQYB+wCnAicXd3wak+2ehW0tyvRiIgkXSYad/+emT0OHAe8A9gQWEaciuZD7n5T9UOsMYVUydMUNCIiQI7hzSmZDFpCMbNPAicT18G53t2/lLafSqxVtQOfKiY9MzsMOBdoBC5w94GtdRUTzUjVaEREIEeiSUs4vxfYgXQfDbFGc5W7v1LN4MzsLcQZond294KZbZK2bw8cnWKaBtxiZtuk034MHAzMBe43s2vd/fFqxrmOQiE+q+lMRAToZjCAmR0IzCEOZ24A5hHvpfkA8GRKBNX0ceBsdy8AuPtLafuRwJXuXnD3p1OMe6XHHHd/yt3bgCsZ6KUM2mKNRvfRiIhE3dVofgic6O6/Ld9hZu8GzgO2q0ZgyTbAm8zsf4kj3L6QliaYDtxTctzctA3g+bLte1cxvvW1FvtolGhERKD7RLMFcH0n+24ALu9rAGZ2C3GodLnTiPFNIN4cuifgZrY1sVZVLqNyDS2rsA0zmwXMAnB3mpubex480NTUtM65hWdHshjYeJMpjOzlNYe68jLXA5W5PtRbmQeqvN0lmnuBM83sdHdfXtxoZmOBr6f9feLuB3W2z8w+DvzW3TPgPjPrAJqJNZXNSg6dQWzWo4vt5Z87G5id3mYtLS29ir+5uZnSc7OXY+vektYCoZfXHOrKy1wPVOb6UG9l7mt5p02bluu47hLNh4ArgBYze4q199FsDTxM7JCvpt8BbwXuSJ39I4EW4Frgl2b2XeJggJnAfcSazkwz2wp4IcV3bJVjXEdWHAygKWhERIDu76N5Fnijmc0kjvBacx+Nuz85APFdCFxoZo8CbcDxqXbzmJk58Dhx2PNJ7t4OYGYnE4djNwIXuvtjAxDnWoWV8Vl9NCIiAIQsq9iFUW+yefMqtrB1q7zq2XHjVWS/vZiGH/mwHXlWb80LoDLXi3orcz81nVXqM19Hrxc+M7MRZnZbb88fttpaIQQYMXKwIxERGRL6ssJmA/Dm/gpk2Ci0wshRhAYtXioiAt0vfPZUF7v1TVpJSjQiIhJ1N+psIvAF4roz5UYCv+/3iGpdQUsEiIiU6i7RPASsdPdby3eY2ShydALVm0yLnomIrKO7RPNNYHkn+9qAas91Vnu06JmIyDq6u4/mji72ZcCd/R1QzVPTmYjIOjrt0C9Oyd8dM5vSf+EMAxoMICKyjq5qNLeb2Z3ApcC97t5R3GFmDcQp+Y8D9gd2rGqUtaTQShg1ZrCjEBEZMrpKNLsRZzeeDWydhjq/Slz8bGvgSeCnwGeqHWRNaStoGWcRkRKdJpq0cNiPgB+Z2WbATsB4YBHwd3d/YWBCrDGt6qMRESnV7VLOAO7+POsuKCYVZFkWR52NVKIRESnS3f39aVUbZBmMVqIRESlSoulPa9aiUaIRESlSoulPa9ai0WAAEZEiJZr+VKzRaDCAiMganQ4GMLNLgW5XRXP34/o1olrW1gowbBc8ExHpja5Gnc0ped0MHA9cBzwLbA4cAVxcvdDAzHYFfgKMJi7Z/Al3v8/MAnAu8DZgBXCCuz+Uzjke+Gq6xJnuXtUY11GIiUY1GhGRtbq6j+YbxddmdhPwdnf/U8m2/YCvVTc8zgG+4e43mtnb0vsDgMOBmemxN3A+sLeZTQS+DuxBrI09aGbXuvuiKscZFRONBgOIiKyRt49mH+Cesm33Am/o33DWkwHj0uuNgXnp9ZHAJe6eufs9wHgzmwocCtzs7gtTcrkZOKzKMa4NtphoNLxZRGSNXDdsAn8DzjKz/3H3lWY2BvgG8HD1QgPi9DY3mdn/EZPiG9P26ax7A+nctK2z7esxs1nEKXZwd5qbm3sVYFNT05pzV44cwVJg4qZTaezl9WpBaZnrhcpcH+qtzANV3ryJ5gTgl8ASM1sETAAeAI7tawBmdguwaYVdpwEHAp9196vMzICfAwdRecG1rIvt63H32cR53ACylpaWnoYOQHNzM8VzO16JzwuXryCE3l2vFpSWuV6ozPWh3src1/JOmzYt13HdJprU8d5AnKV5KjANmO/uz/U6uhLuflAXn30J8On09tfABen1XGCzkkNnEJvV5hL7cEq339EfcebSWryPRk1nIiJF3SYad8/M7B/ARoMw59k84M3EZPFW4ozRANcCJ5vZlcTBAEvcfX4atHCWmU1Ixx0CnDpg0bYVoKEBmkYM2EeKiAx1eQcD/A3YppqBdOK/ge+OlH8GAAAYpUlEQVSY2SPAWaQ+FeAG4CniEOyfAZ8AcPeFwBnA/enxzbRtYKTVNUOo1IInIlKf8vbR3AH8wcwuItZo1vR7uPuF/R/Wmmv/Gdi9wvYMOKmTcy4EqhZTl9oKGtosIlImb6LZF3ia2IxVKmOwvtSHotaVmudMRKRM3vVo3lLtQIaDrK2ggQAiImXy1mjWSKPQ1nRCuHtHv0ZUywpaXVNEpFyuRGNm04nLOu9PXM65VGN/B1WzCq0wZuxgRyEiMqTkHXX2E6CNeAPlMuD1xCHGH6tSXLWp0Ko+GhGRMnkTzRuBD7v7w0Dm7o8AJwKfr1pktajQShg1ZrCjEBEZUvImmnbiNP0Ai81sMrCcTuYRq1ttBdVoRETK5E009xLXfgG4CfgV8FvifGdSpMEAIiLryTvq7IOsTUqfITaZbQR8vxpB1aKso0M3bIqIVJD3PprFJa9XAmdWLaJa1VaIz1qLRkRkHXmHN78C3AXcmR4Pp2lgpKhNq2uKiFSSt49mb+Jw5l2JfTMLzew6M/tC1SKrNYVUo9FgABGRdeRtOptDnCn5F2a2DXAc8Eni0sn/V73wakhaxjloMICIyDryNp19jDih5r7ENWLuAo4B/ly90GpMSjQadSYisq68o87OA/5DXOvl9+4+v3oh1aiC+mhERCrJm2hmEGs0+wOfNrMRpMEB7n5ZtYKrKW2q0YiIVJK3j2YecAVwhZntBrwPOBn4MNCnRGNm7wdOB7YD9nL3B0r2nUqc6qYd+JS735S2HwacS5zQ8wJ3Pztt3wq4EpgIPAR80N3b+hJfXpkGA4iIVJRr1JmZfdbMrjGzhcA1wObAF4jJoa8eBd5DrCGVfub2wNHADsBhwHlm1mhmjcCPgcOB7YFj0rEA3wK+5+4zgUXEJDUwCivjs+Y6ExFZR96ms12A3wGfdfen+jMAd38CwMzKdx0JXOnuBeBpM5sD7JX2zSnGYWZXAkea2RPAW4Fj0zEXE2tK5/dnvJ1SjUZEpKK8TWcnAJhZg5lNHaDBANOBe0rez2XtJJ7Pl23fG5gELHb31RWOX4+ZzQJmAbg7zc3NvQqyqamJ5uZmljU1sBxonjaD0NTj9eRqSrHM9URlrg/1VuaBKm/e4c3jiSPP3gesAsaa2TuJfSpfzXH+LcCmFXad5u7XdHJaqLAto3JzX9bF8RW5+2xgdvG4lpaWzg7tUnNzMy0tLXQsWgiNTbyyeHH3J9W4YpnricpcH+qtzH0t77Rp03Idl/en90+IfR5bAI+nbXcD3wG6TTTuflDOzyk1F9is5P0M4j08dLK9BRhvZk2pVlN6fPW1auZmEZFK8k5BcyBx1Nd8Ui3B3V8GNqlWYMQpb442s1FpNNlM4D7gfmCmmW1lZiOJAwauTXOv3U6sdQEcTxy4MDDalGhERCrJm2iWAOs05JnZ5kCf+2rM7N1mNhd4A3C9md0E4O6PAU6sQf0BOMnd21Nt5WTiujhPxEP9sXS5U4DPpYEDk4Cf9zW+3Apa9ExEpJKQZd1PwmxmXwbeCZwGXE0cWnwWcI27D4c1abJ583rXylZs42z/wTdhySIav/a9fg5t6Km3dmxQmetFvZW5n/poKvWPryNvH823gFbi/SsjgAuBnxJvmhSIU9BoLRoRkfXkHd6cEVfTHA61l+ootMJGGw92FCIiQ07ePpr1mNkRZnZffwZT09rURyMiUkmXNRozm0Jcb2ZX4EngU8Rhw+cRR5z9oNoB1oxCK0EzN4uIrKe7prPzgA3T83uIQ44nAP8LXOzuq6obXg1RH42ISEXdJZo3Aa9196Vm5sDLwE4lw4mlqNCqtWhERCroro9mtLsvBXD3V4AlSjLryzraYfUq3bApIlJBdzWaEWb2IdaOkx5hZh8uPcDdL6xKZLVEMzeLiHSqu0RzL3Bcyfv7gQ+WvM+I99TUN61FIyLSqS4TjbsfMEBx1DbVaEREOtXr+2ikRKEVQMObRUQqUKLpD20x0WgwgIjI+pRo+kOrEo2ISGeUaPqDajQiIp3Kvbi9mW1HXFRsU3c/ycy2BUa6+9+rFl2NyDQYQESkU7lqNGb2fuBOYDprhzdvCHy3SnHVljQYQDMDiIisL2+N5pvAIe7+sJkdlbY9AuzS1wBSEjsd2A7Yy90fSNsPBs4GRgJtwBfd/ba0b3fgImAMcAPwaXfPzGwi8CtgS+AZwNx9UV9j7FYx0WiuMxGR9eTto9mEmFgg3qRZfO5+ec7uPUqcsPOusu0twBHuvhNwPHBpyb7zgVnAzPQ4LG3/MnCru88Ebk3vq29NjUZNZyIi5fLWaB4kNpldUrLtaKDP69G4+xMAZla+/W8lbx8DRpvZKGAiMM7d707nXQK8C7gROBI4IJ1zMXAHcEpfY+xWWyuMGEloaKz6R4mI1Jq8ieZTwB/N7ERgrJndBGwDHFK1yNb1XuBv7l4ws+nA3JJ9c4l9RwBT3H0+gLvPN7NNOrugmc0i1opwd5qbm3sVWFNTE6NDoHX0mF5fo9Y0NTXVTVmLVOb6UG9lHqjy5l3K+Z9plNk7gN8DzwO/d/dlec43s1uATSvsOs3dr+nm3B2Ab7E2qYUKh/W4Cc/dZwOzi+e3tLT09BIANDc307pkEdnIUfT2GrWmubm5bspapDLXh3orc1/LO23atFzH5R7e7O4rAO9NMO5+UG/OM7MZwNXAce7+n7R5LnGVz6IZwLz0eoGZTU21manAS7353J7KCgX1z4iIdCJXojGzP1G51lAgfvH/1t2v68/AzGw8cD1wqrv/pbg9JZFXzWwf1s4u/cO0+1riwIGz03OXtaV+09aqmzVFRDqRd9TZHcQhw3cCl6XnLYAHgAXAhWb2pd4EYGbvNrO5wBuA61P/D8DJwGuBr5nZw+lR7HP5OHABMAf4D3EgAMQEc7CZPQkUh0dXX0GJRkSkM3mbzg4BDi2OEAMws8uBi919bzP7LXAlcE5PA3D3q4nNY+XbzwTO7OScB4AdK2x/BTiwpzH0WaEVxk8a8I8VEakFeWs02wJPlW17FngdgLvfR7zXpj4VCgTVaEREKspbo7kL+IWZ/Q9rO+NPB/4MYGY7AfOrEWBNKLRqMICISCfy1miOT8c+Diwn3kDZCJyQ9rcBx/R3cDVDgwFERDqV9z6ahcDRZtYATAZedveOkv3/qlJ8tUGDAUREOpX7PppkLLABsGVxyhh3L++7qSvZqlXQ3q5EIyLSibz30WwPXE6crTkj3p1fvK+mrif4ygor4wutRSMiUlHePprzgNuJE1ouBSYAPyX23dS1rFVr0YiIdCVvotkFOMXdFwPB3ZcAXwTOqFpkNSJrXRFfqOlMRKSivImmFRiRXreY2ebp3Lq/SzFLa9HoPhoRkcryJpo/AcUFY35DnPLlTuC2agRVS7LWYh+NEo2ISCV5hzeXrkr2FeKqmBux7kJodWlNH40SjYhIRd0mGjNrJC6LfKi7F9L9M5dVPbIaoRqNiEjXum06c/d2YKs8x9ajNcObNQWNiEhFeW/Y/AZwvpl9nTjX2Zq1aUpnCKhHajoTEela3kRzQXr+YMm24k2bumETYNSYwQ1ERGSIyptotqpqFDUsW7kSQoCRIwc7FBGRISnvqLNnAdKkmlPcvd+WBDCz9xOXHNgO2Cstala6f3PirNGnu/v/pW2HAecSa1MXuPvZaftWxAXYJgIPAR9097b+irWSrLASRo4ihFDNjxERqVm5OvjNbLyZ/ZJ44+actO2dZlZxBcweehR4D3HNm0q+x9qlmouj4H4MHA5sDxyT5mID+BbwPXefCSwCTuyH+LqUtWotGhGRruQdSfYTYAmwBXHtGYC7gaP6GoC7P9HZMgNm9i7iyp6PlWzeC5jj7k+l2sqVwJFmFoC3Em8oBbgYeFdf4+tOVlgJo9U/IyLSmbx9NAcC09x9lZllAO7+splVbflmMxsLnAIcDHyhZNd04PmS93OBvYnT4Sx299Ul26d3cf1ZwCwAd6e5ublXcS5pK9C0wVgm9fL8WtTU1NTrv1etUpnrQ72VeaDKmzfRLAGaKVmuOfWd5OqrMbNbgE0r7DrN3a/p5LRvEJvBlhXXvkkqdYZkXWyvyN1nA7OLx7W0tHR2aJcaVyxndWMTvT2/FjU3N9dVeUFlrhf1Vua+lnfatGm5juvJ8OarzOw0oMHM3gCcRWxS65a7H5Tzc0rtDbzPzM4BxgMdZtYKPAhsVnLcDGAe0AKMN7OmVKspbq+qTKtrioh0KW+i+RZxIMCPibM4X0hcj+bcKsWFu7+p+NrMTgeWufuPzKwJmJlGmL0AHA0c6+6Zmd0OvI/Yb3M80Fltqd9kra0wfmK1P0ZEpGblHd6cAd9Pj35lZu8GfghMBq43s4fd/dAuYlltZicDNxGHN1/o7sXBAqcAV6bRcH8Dft7f8ZbLCisJWvRMRKRTIcs67cZYw8weIU6keYW7z616VAMvmzevd61s2Zc+RLbj7jQcd3I/hzR01Vs7NqjM9aLeytxPfTTd3kSYt+nsdOAY4Otm9iDwS+DX7r6wtwEOF1nrSvXRiIh0Idd9NO5+dVqTZiqxf+bdwPNmdm01gxvqsizTYAARkW70aOp/d3+VWJs5H7gHeFs1gqoZq1dBR4cSjYhIF3I1nZXcdX8ssTbzLDHhnFC1yGpBcYkADQYQEelU3j6aecAy4rDhfd39ieqFVEPaimvRaK4zEZHO5E0073L3e8s3mllDXS98VigmGs11JiLSmbz30ayTZMxsJ+INkccC+eYgGI4KBQCCajQiIp3KW6PBzCYTE8vxwC7An4BPVymu2rBmdU310YiIdKbLRGNmI4B3Ejv9DyWuRXMFcbkAc/eXqh3gkJZqNBoMICLSue6GNy8gzmn2L2Afd9/e3c9g7Zo0dS0rDgYYrUQjItKZ7hLN34kzJ+8N7GlmE6ofUg0paHiziEh3ukw07n4A8Brgj8TFx140s+uAscRZnOtbQcObRUS60+3MAO7+rLuf4e4ziSttzgc6gEfSWjH1a02iUY1GRKQzPZ2C5s/uPou4WuYngZ2qElWtKBSgoRGaVLkTEelM7uHNpdy9lTj67Ir+DafGtLUSRo8mhG5nyRYRqVs9qtFImUIrQbMCiIh0qVc1mv5kZu8nrnezHbCXuz9Qsm9n4vDqccR+oT3dvdXMdgcuAsYANwCfTks5TwR+BWwJPEO812dR1YIvxBqNiIh0bijUaB4F3gPcVbrRzJqIq3p+zN13AA4AVqXd5wOzgJnpcVja/mXg1jRw4db0vmqyQithtGo0IiJdGfQaTXEmaDMr33UI8Hd3fyQd90o6biowzt3vTu8vAd4F3AgcSUxIABcDdwCnVCv2sPXrGBmgUK0PEBEZBgY90XRhGyAzs5uAycCV7n4OMB2YW3Lc3LQNYIq7zwdw9/lmtklnFzezWcRaEe5Oc3NzzyM87uM0NTWxevXqnp9bw5qamnr396phKnN9qLcyD1R5ByTRmNktxCHR5U5z92s6Oa0J2A/YE1gB3GpmDwJLKxyb9TQmd58NzC6e39LS0tNLANDc3Exvz61VKnN9UJmHv76Wd9q0fJP3D0iicfeDenHaXOBOd28BMLMbgNcT+21mlBw3g7gwG8ACM5uaajNTgfqe9FNEZAgYCoMBOnMTsLOZbZAGBrwZeDw1jb1qZvukJaaPA4q1omuJyxiQnjurLYmIyAAZ9ERjZu82s7nAG4DrU58MaVjyd4H7gYeBh9z9+nTax4ELiMsW/Ic4EADgbOBgM3sSODi9FxGRQRSyrMfdG8NRNm/evO6PqqDe2nRBZa4XKvPw1099NN1OjTLoNRoRERnelGhERKSqlGhERKSq1EcT6Y8gItI76qPJKfT2kW4i7fX5tfhQmevjoTIP/0c/lbdbSjQiIlJVSjQiIlJVSjR9N7v7Q4Ydlbk+qMzD34CUV4MBRESkqlSjERGRqlKiERGRqhrKC58NeWZ2GHAu0Ahc4O41OYmnmV0IvAN4yd13TNsmAr8CtgSeAczdF6UZs88F3kZcJ+gEd38onXM88NV02TPd/eKBLEdPmNlmwCXEdZI6gNnufu5wLreZjSYumT6K+P/+b9z962a2FXAlMBF4CPigu7eZ2Sji32h34BXgKHd/Jl3rVOBEoB34lLvfNNDlycvMGoEHgBfc/R3DvbwAZvYM8Cox3tXuvsdg/ttWjaaX0j/eHwOHA9sDx5jZ9oMbVa9dBBxWtu3LwK3uPhO4Nb2HWN6Z6TELOB/WJKavA3sDewFfN7MJVY+891YDn3f37YB9gJPSf7/hXO4C8FZ33wXYFTjMzPYBvgV8L5V5EfELlfS8yN1fC3wvHUf6Ox0N7ED8d3Ne+v9hqPo08ETJ++Fe3qK3uPuu7r5Hej9o/7aVaHpvL2COuz/l7m3EX0hHDnJMveLudwELyzYfCRR/vVwMvKtk+yXunrn7PcD4tMjcocDN7r4wLfFwM+snryHD3ecXf7W5+6vEL6LpDONyp9iXpbcj0iMD3gr8Jm0vL3Pxb/Eb4MD06/dI4tLqBXd/mrhcx14DUIQeM7MZwNuJy4qQ4h+25e3GoP3bVqLpvenA8yXv56Ztw8WUtMgc6XmTtL2zctfs38PMtgR2A+5lmJfbzBrN7GHi6rM3E9dzWuzuq9MhpfGvKVvavwSYRG2V+fvAl4jNoxDjH87lLcqAP5rZg2Y2K20btH/bSjS9V2nqhXoYK95ZuWvy72FmGwJXAZ9x96VdHDosyu3u7e6+K3EJ9L2A7SocVoy/pstsZsV+xwdLNncVe02Xt8y+7v56YrPYSWa2fxfHVr3cSjS9NxfYrOT9DKB3q6cNTQtS9Zn0/FLa3lm5a+7vYWYjiEnmcnf/bdo87MsN4O6LgTuI/VPj03LpsG78a8qW9m9MbGKtlTLvC7wzdYxfSWwy+z7Dt7xruPu89PwScDXxR8Wg/dtWoum9+4GZZraVmY0kdhZeO8gx9adrgePT6+OBa0q2H2dmIXUkL0nV8JuAQ8xsQuowPCRtG5JS2/vPgSfc/bslu4Ztuc1sspmNT6/HAAcR+6ZuB96XDisvc/Fv8T7gNnfP0vajzWxUGsE1E7hvYEqRn7uf6u4z3H1L4v+ft7n7Bxim5S0ys7FmtlHxNfHf5KMM4r9tJZpeSm24JxP/8E/ETf7Y4EbVO2Z2BXA38Dozm2tmJwJnAweb2ZPAwek9wA3AU8QO0Z8BnwBw94XAGcQEfD/wzbRtqNoX+CDwVjN7OD3exvAu91TgdjP7OzHWm93998ApwOfMbA6xT+Ln6fifA5PS9s+RRimlf+cOPA78ATjJ3dsHtCR9M9zLOwX4s5k9QkyI17v7HxjEf9uagkZERKpKNRoREakqJRoREakqJRoREakqJRoREakqJRoREakqJRqRAWZmN6ZZcfvzmqeb2WX9eU2R/qJlAkR6Kd1xPoU4FXvRRe5+clfnufvh1YxLZKhRohHpmyPc/ZbBDkJkKFOiEelnZnYC8N/ERbWOA+YT7ya/Ne2/A7jM3S8ws9cS70jfFVhFXC/kqHTcG4kLUm0D/Bv4tLv/Ne3biriO0OuBe4B/lcWwD/Bd4lpJz6Zz7yiJ73+AyUAL8FV3v7zf/xAiifpoRKpjb+K0Hs3ExaN+mxaSKncG8EdgAnHSwh/CmkWnrgd+QJwm5bvA9WY2KZ33S+DBdP0zWDuHFWY2PZ17JnEVyS8AV6W5zsamax7u7hsBbwQe7r9ii6xPNRqRvvmdma0uef9FYs3kJeD7aVLGX5nZ54kLcF1adv4qYAtgmrvPBf6ctr8deNLdi8dfYWafAo4ws9uAPYGD3L0A3GVm15Vc87+AG9z9hvT+ZjN7gLhU72+Ia7PsaGbPpckT5/f1jyDSFSUakb55V3kfTWqaeiElmaJngWkVzv8SsUZyn5ktAr7j7hemY58tO/ZZ4sJT04hLDi8v21ec0n0L4P1mdkTJ/hHA7e6+3MyOItZyfm5mfyEuaf3P3CUW6SElGpHqmG5moSTZbE6FZSTc/UVifw5mth9wi5ndRVz3Y4uywzcnzh48H5hgZmNLks3mrF2U6nngUnf/70qBuftNwE1pqYAziTP2vql3xRTpnhKNSHVsAnzKzM4jrs2+HXE69nWY2fuBu1Oz2SJismhPx/7QzI4lTlH/XmLH/u/dvSU1hX3DzL5CXNTqCNYmssuA+83sUOAWYm1mH+I08KuI/Ue3AiuBZaw7PFuk3ynRiPTNdWZW+kV9M3FBqXuJC2S1AAuA97n7KxXO3xP4vpltnI77tLs/DWuWIj4XOJ+YJN7h7i3pvGOBi4krQN4NXAKMB3D3583sSOAc4ApiIrkP+DhxANDniX1FGXEgwCf6/mcQ6ZzWoxHpZ6mP5iPuvt9gxyIyFGh4s4iIVJUSjYiIVJWazkREpKpUoxERkapSohERkapSohERkapSohERkapSohERkar6/8It3Iib/asUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12291a358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.linspace(0, 5000, len(avg_score)), np.asarray(avg_score))\n",
    "plt.xlabel('Episodes')\n",
    "plt.ylabel('Average Reward (100 episode)')\n",
    "plt.title('SARSA Agent Training')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Try it out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "last reward: -1, total score: -13\n"
     ]
    }
   ],
   "source": [
    "# Use policy to play\n",
    "state = env.reset()\n",
    "score = 0\n",
    "while True:\n",
    "    action = np.argmax(Q_sarsa[state])\n",
    "    state, reward, done, info = env.step(action)\n",
    "    score += reward\n",
    "    if done:\n",
    "        print('last reward: {}, total score: {}'.format(reward, score))\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py3",
   "language": "python",
   "name": "py3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}