mountaincar-a2c-Copy1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Whatami\n",
    "\n",
    "I am a simple experiment on using qlearning agent setup for MountainCar problem.\n",
    "Being off-policy value based algorithm, qlearning has comparatively poor convergence on this problem (see a2c nearby for comparison) yet it does manage to find some policy that brings him to the end.\n",
    "\n",
    "\n",
    "## About OpenAI Gym\n",
    "\n",
    "* Its a recently published platform that basicly allows you to train agents in a wide variety of environments with near-identical interface.\n",
    "* This is twice as awesome since now we don't need to write a new wrapper for every game\n",
    "* Go check it out!\n",
    "  * Blog post - https://openai.com/blog/openai-gym-beta/\n",
    "  * Github - https://github.com/openai/gym\n",
    "\n",
    "\n",
    "## New to Lasagne and AgentNet?\n",
    "* We only require surface level knowledge of theano and lasagne, so you can just learn them as you go.\n",
    "* Alternatively, you can find Lasagne tutorials here:\n",
    " * Official mnist example: http://lasagne.readthedocs.io/en/latest/user/tutorial.html\n",
    " * From scratch: https://github.com/ddtm/dl-course/tree/master/Seminar4\n",
    " * From theano: https://github.com/craffel/Lasagne-tutorial/blob/master/examples/tutorial.ipynb\n",
    "* This is pretty much the basic tutorial for AgentNet, so it's okay not to know it.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiment setup\n",
    "* Here we basically just load the game and check that it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.\n",
      "  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: THEANO_FLAGS=\"floatX=float32\"\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function \n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "%env THEANO_FLAGS=\"floatX=float32\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#global params.\n",
    "GAME = \"MountainCar-v0\"\n",
    "\n",
    "#number of parallel agents and batch sequence length (frames)\n",
    "N_AGENTS = 1\n",
    "SEQ_LENGTH = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:22:35,789] Making new env: MountainCar-v0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.5693506  -0.00066286]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import gym\n",
    "env = gym.make(GAME)\n",
    "obs = env.step(0)[0]\n",
    "action_names = np.array([\"left\",'stop',\"right\"]) #i guess so... i may be wrong\n",
    "state_size = len(obs)\n",
    "print(obs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic agent setup\n",
    "Here we define a simple agent that maps game images into Qvalues using shallow neural network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Couldn't import dot_parser, loading of dot files will not be possible.\n"
     ]
    }
   ],
   "source": [
    "import lasagne\n",
    "from lasagne.layers import InputLayer,DenseLayer,batch_norm,dropout\n",
    "#image observation at current tick goes here, shape = (sample_i,x,y,color)\n",
    "observation_layer = InputLayer((None,state_size))\n",
    "\n",
    "dense0 = DenseLayer(observation_layer,100,name='dense1')\n",
    "dense1 = DenseLayer(dense0,256,name='dense2')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#a layer that predicts Qvalues\n",
    "qvalues_layer = DenseLayer(dense1,\n",
    "                   num_units = env.action_space.n,\n",
    "                   nonlinearity=lasagne.nonlinearities.linear,\n",
    "                   name=\"q-evaluator layer\")\n",
    "\n",
    "#To pick actions, we use an epsilon-greedy resolver (epsilon is a property)\n",
    "from agentnet.resolver import EpsilonGreedyResolver\n",
    "action_layer = EpsilonGreedyResolver(qvalues_layer,name=\"e-greedy action picker\")\n",
    "\n",
    "action_layer.epsilon.set_value(np.float32(0.05))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Finally, agent\n",
    "We declare that this network is and MDP agent with such and such inputs, states and outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from agentnet.agent import Agent\n",
    "#all together\n",
    "agent = Agent(observation_layers=observation_layer,\n",
    "              policy_estimators=qvalues_layer,\n",
    "              action_layers=action_layer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[dense1.W,\n",
       " dense1.b,\n",
       " dense2.W,\n",
       " dense2.b,\n",
       " q-evaluator layer.W,\n",
       " q-evaluator layer.b]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Since it's a single lasagne network, one can get it's weights, output, etc\n",
    "weights = lasagne.layers.get_all_params(action_layer,trainable=True)\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create and manage a pool of atari sessions to play with\n",
    "\n",
    "* To make training more stable, we shall have an entire batch of game sessions each happening independent of others\n",
    "* Why several parallel agents help training: http://arxiv.org/pdf/1602.01783v1.pdf\n",
    "* Alternative approach: store more sessions: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:22:36,589] Making new env: MountainCar-v0\n"
     ]
    }
   ],
   "source": [
    "from agentnet.experiments.openai_gym.pool import EnvPool\n",
    "\n",
    "pool = EnvPool(agent,GAME, N_AGENTS,max_size=10000)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['right' 'right' 'right' 'right' 'right' 'right' 'left']]\n",
      "[[-1. -1. -1. -1. -1. -1.  0.]]\n",
      "CPU times: user 7.81 ms, sys: 31.7 ms, total: 39.5 ms\n",
      "Wall time: 17.7 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#interact for 7 ticks\n",
    "_,action_log,reward_log,_,_,_  = pool.interact(7)\n",
    "\n",
    "\n",
    "print(action_names[action_log])\n",
    "print(reward_log)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#load first sessions (this function calls interact and remembers sessions)\n",
    "pool.update(SEQ_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-learning\n",
    "* An agent has a method that produces symbolic environment interaction sessions\n",
    "* Such sessions are in sequences of observations, agent memory, actions, q-values,etc\n",
    "  * one has to pre-define maximum session length.\n",
    "\n",
    "* SessionPool also stores rewards (Q-learning objective)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#get agent's Qvalues obtained via experience replay\n",
    "replay = pool.experience_replay.sample_session_batch(100,replace=True)\n",
    "\n",
    "_,_,_,_,qvalues_seq = agent.get_sessions(\n",
    "    replay,\n",
    "    session_length=SEQ_LENGTH,\n",
    "    optimize_experience_replay=True,\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#get reference Qvalues according to Qlearning algorithm\n",
    "from agentnet.learning import qlearning\n",
    "\n",
    "#crop rewards to [-1,+1] to avoid explosion.\n",
    "#import theano.tensor as T\n",
    "#rewards = T.maximum(-1,T.minimum(rewards,1))\n",
    "\n",
    "#loss for Qlearning = (Q(s,a) - (r+gamma*Q(s',a_max)))^2\n",
    "\n",
    "elwise_mse_loss = qlearning.get_elementwise_objective(qvalues_seq,\n",
    "                                                      replay.actions[0],\n",
    "                                                      replay.rewards,\n",
    "                                                      replay.is_alive,\n",
    "                                                      gamma_or_gammas=0.99,)\n",
    "\n",
    "#compute mean over \"alive\" fragments\n",
    "loss = elwise_mse_loss.sum() / replay.is_alive.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Compute weight updates\n",
    "updates = lasagne.updates.rmsprop(loss,weights,learning_rate=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#compile train function\n",
    "import theano\n",
    "train_step = theano.function([],loss,updates=updates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:22:42,680] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:22:42,683] Clearing 3 monitor files from previous run (because force=True was provided)\n",
      "[2016-11-28 01:22:42,760] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 200 timesteps with reward=-200.0\n"
     ]
    }
   ],
   "source": [
    "#for MountainCar-v0 evaluation session is cropped to 200 ticks\n",
    "untrained_reward = pool.evaluate(save_path=\"./records\",record_video=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "#video_path=\"./records/openaigym.video.0.7346.video000000.mp4\"\n",
    "\n",
    "#HTML(\"\"\"\n",
    "#<video width=\"640\" height=\"480\" controls>\n",
    "#  <source src=\"{}\" type=\"video/mp4\">\n",
    "#</video>\n",
    "#\"\"\".format(video_path))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#starting epoch\n",
    "epoch_counter = 1\n",
    "\n",
    "#full game rewards\n",
    "rewards = {epoch_counter:untrained_reward}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [00:03<00:00, 254.12it/s]\n"
     ]
    }
   ],
   "source": [
    "#pre-fill pool\n",
    "from tqdm import tqdm\n",
    "for i in tqdm(range(1000)):\n",
    "    pool.update(SEQ_LENGTH,append=True,)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  1%|          | 100/10000 [00:26<1:00:37,  2.72it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=100\tepsilon=0.050\treward/step=-0.99994\tpool_size=2001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  2%|▏         | 200/10000 [01:04<51:11,  3.19it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=200\tepsilon=0.050\treward/step=-0.99980\tpool_size=3001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  3%|▎         | 300/10000 [01:27<36:27,  4.43it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=300\tepsilon=0.050\treward/step=-0.99935\tpool_size=4001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  4%|▍         | 400/10000 [01:49<34:10,  4.68it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=400\tepsilon=0.050\treward/step=-0.99909\tpool_size=5001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  5%|▍         | 499/10000 [02:11<35:31,  4.46it/s][2016-11-28 01:24:59,030] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:24:59,033] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=500\tepsilon=0.050\treward/step=-0.99888\tpool_size=6001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:24:59,607] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      "  5%|▌         | 500/10000 [02:12<1:02:11,  2.55it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -130.000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  6%|▌         | 600/10000 [02:35<34:07,  4.59it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=600\tepsilon=0.050\treward/step=-0.99865\tpool_size=7001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  7%|▋         | 700/10000 [02:57<32:29,  4.77it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=700\tepsilon=0.050\treward/step=-0.99848\tpool_size=8001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  8%|▊         | 800/10000 [03:19<32:04,  4.78it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=800\tepsilon=0.050\treward/step=-0.99830\tpool_size=9001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  9%|▉         | 901/10000 [03:40<29:06,  5.21it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=900\tepsilon=0.050\treward/step=-0.99816\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 10%|▉         | 999/10000 [04:00<34:53,  4.30it/s][2016-11-28 01:26:47,986] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:26:47,989] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=1000\tepsilon=0.050\treward/step=-0.99786\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:26:48,522] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 10%|█         | 1001/10000 [04:01<52:44,  2.84it/s]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -134.400\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 11%|█         | 1100/10000 [04:24<31:49,  4.66it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=1100\tepsilon=0.050\treward/step=-0.99757\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 12%|█▏        | 1201/10000 [04:48<31:32,  4.65it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=1200\tepsilon=0.050\treward/step=-0.99727\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 13%|█▎        | 1301/10000 [05:08<27:51,  5.20it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=1300\tepsilon=0.050\treward/step=-0.99711\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 14%|█▍        | 1400/10000 [05:29<34:56,  4.10it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=1400\tepsilon=0.050\treward/step=-0.99696\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 15%|█▍        | 1499/10000 [05:50<31:23,  4.51it/s][2016-11-28 01:28:37,308] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:28:37,312] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=1500\tepsilon=0.050\treward/step=-0.99680\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:28:37,775] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 15%|█▌        | 1500/10000 [05:50<51:19,  2.76it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -107.500\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 1600/10000 [06:11<31:32,  4.44it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=1600\tepsilon=0.050\treward/step=-0.99671\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 17%|█▋        | 1700/10000 [06:40<45:40,  3.03it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=1700\tepsilon=0.050\treward/step=-0.99659\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 18%|█▊        | 1800/10000 [07:12<38:45,  3.53it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=1800\tepsilon=0.050\treward/step=-0.99654\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 19%|█▉        | 1900/10000 [07:45<49:13,  2.74it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=1900\tepsilon=0.050\treward/step=-0.99649\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 20%|█▉        | 1999/10000 [08:14<39:04,  3.41it/s][2016-11-28 01:31:02,171] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:31:02,174] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=2000\tepsilon=0.050\treward/step=-0.99645\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:31:02,868] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 20%|██        | 2000/10000 [08:15<1:09:05,  1.93it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -162.600\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 21%|██        | 2100/10000 [08:47<52:01,  2.53it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=2100\tepsilon=0.050\treward/step=-0.99643\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 22%|██▏       | 2200/10000 [09:14<25:11,  5.16it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=2200\tepsilon=0.050\treward/step=-0.99645\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 23%|██▎       | 2300/10000 [09:34<26:04,  4.92it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=2300\tepsilon=0.050\treward/step=-0.99647\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 24%|██▍       | 2401/10000 [09:55<24:13,  5.23it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=2400\tepsilon=0.050\treward/step=-0.99649\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 25%|██▍       | 2499/10000 [10:15<26:48,  4.66it/s][2016-11-28 01:33:02,298] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:33:02,301] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=2500\tepsilon=0.050\treward/step=-0.99653\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:33:02,768] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 25%|██▌       | 2500/10000 [10:15<44:11,  2.83it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -121.800\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 26%|██▌       | 2601/10000 [10:36<23:59,  5.14it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=2600\tepsilon=0.050\treward/step=-0.99657\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 27%|██▋       | 2700/10000 [10:58<26:32,  4.58it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=2700\tepsilon=0.050\treward/step=-0.99663\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 28%|██▊       | 2801/10000 [11:19<24:36,  4.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=2800\tepsilon=0.050\treward/step=-0.99666\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 29%|██▉       | 2901/10000 [11:40<22:09,  5.34it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=2900\tepsilon=0.050\treward/step=-0.99668\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 30%|██▉       | 2999/10000 [12:00<24:11,  4.82it/s][2016-11-28 01:34:47,929] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:34:47,931] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=3000\tepsilon=0.050\treward/step=-0.99668\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:34:48,381] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 30%|███       | 3000/10000 [12:01<42:19,  2.76it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -124.100\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 31%|███       | 3100/10000 [12:22<25:55,  4.44it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=3100\tepsilon=0.050\treward/step=-0.99664\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 32%|███▏      | 3201/10000 [12:44<22:33,  5.02it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=3200\tepsilon=0.050\treward/step=-0.99664\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 33%|███▎      | 3301/10000 [13:04<21:38,  5.16it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=3300\tepsilon=0.050\treward/step=-0.99666\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 34%|███▍      | 3401/10000 [13:25<21:30,  5.11it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=3400\tepsilon=0.050\treward/step=-0.99666\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 35%|███▍      | 3499/10000 [13:46<22:45,  4.76it/s][2016-11-28 01:36:33,832] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:36:33,835] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=3500\tepsilon=0.050\treward/step=-0.99666\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:36:34,314] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 35%|███▌      | 3500/10000 [13:47<38:39,  2.80it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -126.000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 36%|███▌      | 3600/10000 [14:10<22:12,  4.80it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=3600\tepsilon=0.050\treward/step=-0.99669\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 37%|███▋      | 3700/10000 [14:32<25:20,  4.14it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=3700\tepsilon=0.050\treward/step=-0.99670\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 38%|███▊      | 3800/10000 [14:54<23:51,  4.33it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=3800\tepsilon=0.050\treward/step=-0.99668\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 39%|███▉      | 3900/10000 [15:16<21:07,  4.81it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=3900\tepsilon=0.050\treward/step=-0.99669\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 40%|███▉      | 3999/10000 [15:38<25:30,  3.92it/s][2016-11-28 01:38:26,087] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:38:26,090] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=4000\tepsilon=0.050\treward/step=-0.99674\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:38:26,664] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 40%|████      | 4001/10000 [15:39<35:29,  2.82it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -153.700\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 41%|████      | 4100/10000 [16:00<22:13,  4.42it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=4100\tepsilon=0.050\treward/step=-0.99682\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 42%|████▏     | 4200/10000 [16:21<20:06,  4.81it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=4200\tepsilon=0.050\treward/step=-0.99685\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 43%|████▎     | 4300/10000 [16:43<21:43,  4.37it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=4300\tepsilon=0.050\treward/step=-0.99688\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 44%|████▍     | 4400/10000 [17:05<20:24,  4.57it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=4400\tepsilon=0.050\treward/step=-0.99692\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 45%|████▍     | 4499/10000 [17:29<26:10,  3.50it/s][2016-11-28 01:40:16,564] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:40:16,567] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=4500\tepsilon=0.050\treward/step=-0.99697\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:40:17,177] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 45%|████▌     | 4500/10000 [17:30<41:43,  2.20it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -154.800\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 46%|████▌     | 4600/10000 [17:54<21:20,  4.22it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=4600\tepsilon=0.050\treward/step=-0.99697\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 47%|████▋     | 4700/10000 [18:17<20:50,  4.24it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=4700\tepsilon=0.050\treward/step=-0.99700\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 48%|████▊     | 4800/10000 [18:40<18:29,  4.69it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=4800\tepsilon=0.050\treward/step=-0.99702\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 49%|████▉     | 4900/10000 [19:03<22:02,  3.86it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=4900\tepsilon=0.050\treward/step=-0.99705\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 50%|████▉     | 4999/10000 [19:27<18:19,  4.55it/s][2016-11-28 01:42:14,252] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:42:14,256] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=5000\tepsilon=0.050\treward/step=-0.99707\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:42:15,107] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 50%|█████     | 5000/10000 [19:28<40:06,  2.08it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -200.000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 51%|█████     | 5100/10000 [19:50<17:11,  4.75it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=5100\tepsilon=0.050\treward/step=-0.99707\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 52%|█████▏    | 5200/10000 [20:10<15:15,  5.24it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=5200\tepsilon=0.050\treward/step=-0.99711\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 53%|█████▎    | 5301/10000 [20:32<21:05,  3.71it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=5300\tepsilon=0.050\treward/step=-0.99712\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 54%|█████▍    | 5401/10000 [20:52<14:40,  5.22it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=5400\tepsilon=0.050\treward/step=-0.99711\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 55%|█████▍    | 5499/10000 [21:12<14:38,  5.12it/s][2016-11-28 01:43:59,827] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:43:59,830] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=5500\tepsilon=0.050\treward/step=-0.99709\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:44:00,452] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 55%|█████▌    | 5500/10000 [21:13<28:44,  2.61it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -163.000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 56%|█████▌    | 5601/10000 [21:36<14:38,  5.01it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=5600\tepsilon=0.050\treward/step=-0.99707\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 57%|█████▋    | 5700/10000 [21:57<14:57,  4.79it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=5700\tepsilon=0.050\treward/step=-0.99709\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 58%|█████▊    | 5801/10000 [22:19<16:29,  4.24it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=5800\tepsilon=0.050\treward/step=-0.99712\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 59%|█████▉    | 5901/10000 [22:40<14:08,  4.83it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=5900\tepsilon=0.050\treward/step=-0.99714\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 60%|█████▉    | 5999/10000 [23:01<15:24,  4.33it/s][2016-11-28 01:45:48,193] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:45:48,196] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=6000\tepsilon=0.050\treward/step=-0.99711\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:45:48,767] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 60%|██████    | 6000/10000 [23:01<26:18,  2.53it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -129.700\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 61%|██████    | 6100/10000 [23:25<16:10,  4.02it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=6100\tepsilon=0.050\treward/step=-0.99706\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 62%|██████▏   | 6200/10000 [23:46<12:58,  4.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=6200\tepsilon=0.050\treward/step=-0.99697\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 63%|██████▎   | 6301/10000 [24:07<12:30,  4.93it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=6300\tepsilon=0.050\treward/step=-0.99687\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 64%|██████▍   | 6401/10000 [24:28<11:48,  5.08it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=6400\tepsilon=0.050\treward/step=-0.99683\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 65%|██████▍   | 6499/10000 [24:49<13:19,  4.38it/s][2016-11-28 01:47:36,500] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:47:36,504] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=6500\tepsilon=0.050\treward/step=-0.99685\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:47:36,949] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 65%|██████▌   | 6500/10000 [24:50<22:11,  2.63it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -113.500\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 66%|██████▌   | 6601/10000 [25:11<10:45,  5.26it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=6600\tepsilon=0.050\treward/step=-0.99685\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 67%|██████▋   | 6700/10000 [25:32<12:43,  4.32it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=6700\tepsilon=0.050\treward/step=-0.99678\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 68%|██████▊   | 6800/10000 [25:54<11:37,  4.59it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=6800\tepsilon=0.050\treward/step=-0.99673\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 69%|██████▉   | 6900/10000 [26:16<10:18,  5.01it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=6900\tepsilon=0.050\treward/step=-0.99668\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 70%|██████▉   | 6999/10000 [26:39<12:57,  3.86it/s][2016-11-28 01:49:26,442] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:49:26,456] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=7000\tepsilon=0.050\treward/step=-0.99666\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:49:27,272] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 70%|███████   | 7000/10000 [26:40<25:22,  1.97it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -190.300\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 71%|███████   | 7100/10000 [27:02<11:28,  4.21it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=7100\tepsilon=0.050\treward/step=-0.99667\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 72%|███████▏  | 7201/10000 [27:25<10:30,  4.44it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=7200\tepsilon=0.050\treward/step=-0.99676\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 73%|███████▎  | 7301/10000 [27:47<09:52,  4.55it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=7300\tepsilon=0.050\treward/step=-0.99685\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 74%|███████▍  | 7401/10000 [28:08<08:26,  5.14it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=7400\tepsilon=0.050\treward/step=-0.99688\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 75%|███████▍  | 7499/10000 [28:29<08:24,  4.95it/s][2016-11-28 01:51:17,198] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:51:17,201] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=7500\tepsilon=0.050\treward/step=-0.99688\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:51:17,664] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 75%|███████▌  | 7500/10000 [28:30<15:04,  2.76it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -119.400\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 76%|███████▌  | 7601/10000 [28:54<08:40,  4.61it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=7600\tepsilon=0.050\treward/step=-0.99690\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 77%|███████▋  | 7700/10000 [29:15<08:09,  4.70it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=7700\tepsilon=0.050\treward/step=-0.99695\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 78%|███████▊  | 7800/10000 [29:38<07:37,  4.81it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=7800\tepsilon=0.050\treward/step=-0.99699\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 79%|███████▉  | 7901/10000 [29:59<07:30,  4.66it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=7900\tepsilon=0.050\treward/step=-0.99703\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 80%|███████▉  | 7999/10000 [30:20<07:56,  4.20it/s][2016-11-28 01:53:07,570] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:53:07,573] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=8000\tepsilon=0.050\treward/step=-0.99711\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:53:08,338] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 80%|████████  | 8001/10000 [30:21<12:23,  2.69it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -200.000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 81%|████████  | 8100/10000 [30:43<06:40,  4.74it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=8100\tepsilon=0.050\treward/step=-0.99716\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 82%|████████▏ | 8200/10000 [31:04<06:04,  4.94it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=8200\tepsilon=0.050\treward/step=-0.99718\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 83%|████████▎ | 8300/10000 [31:27<06:59,  4.05it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=8300\tepsilon=0.050\treward/step=-0.99718\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 84%|████████▍ | 8400/10000 [31:51<05:53,  4.52it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=8400\tepsilon=0.050\treward/step=-0.99723\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 85%|████████▍ | 8499/10000 [32:13<05:29,  4.56it/s][2016-11-28 01:55:00,633] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:55:00,635] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=8500\tepsilon=0.050\treward/step=-0.99725\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:55:01,376] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 85%|████████▌ | 8500/10000 [32:14<10:57,  2.28it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -185.300\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 86%|████████▌ | 8601/10000 [32:35<04:39,  5.01it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=8600\tepsilon=0.050\treward/step=-0.99726\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 87%|████████▋ | 8701/10000 [32:55<04:11,  5.17it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=8700\tepsilon=0.050\treward/step=-0.99726\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 88%|████████▊ | 8801/10000 [33:16<04:07,  4.84it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=8800\tepsilon=0.050\treward/step=-0.99728\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 89%|████████▉ | 8900/10000 [33:36<04:36,  3.97it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=8900\tepsilon=0.050\treward/step=-0.99727\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 90%|████████▉ | 8999/10000 [33:56<03:27,  4.83it/s][2016-11-28 01:56:43,446] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:56:43,450] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=9000\tepsilon=0.050\treward/step=-0.99721\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:56:43,926] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 90%|█████████ | 9001/10000 [33:57<04:54,  3.39it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -116.700\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 91%|█████████ | 9101/10000 [34:17<02:46,  5.38it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=9100\tepsilon=0.050\treward/step=-0.99718\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 92%|█████████▏| 9200/10000 [34:38<03:02,  4.39it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=9200\tepsilon=0.050\treward/step=-0.99718\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 93%|█████████▎| 9300/10000 [34:58<02:15,  5.16it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=9300\tepsilon=0.050\treward/step=-0.99716\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 94%|█████████▍| 9401/10000 [35:18<02:00,  4.98it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=9400\tepsilon=0.050\treward/step=-0.99714\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 95%|█████████▍| 9499/10000 [35:39<01:48,  4.63it/s][2016-11-28 01:58:26,504] Making new env: MountainCar-v0\n",
      "[2016-11-28 01:58:26,507] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=9500\tepsilon=0.050\treward/step=-0.99714\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 01:58:26,958] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      " 95%|█████████▌| 9501/10000 [35:40<02:29,  3.34it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -122.000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 96%|█████████▌| 9600/10000 [36:00<01:29,  4.49it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=9600\tepsilon=0.050\treward/step=-0.99713\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 97%|█████████▋| 9701/10000 [36:20<00:55,  5.36it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=9700\tepsilon=0.050\treward/step=-0.99712\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|█████████▊| 9800/10000 [36:41<00:42,  4.69it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=9800\tepsilon=0.050\treward/step=-0.99707\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 99%|█████████▉| 9901/10000 [37:02<00:19,  5.08it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=9900\tepsilon=0.050\treward/step=-0.99707\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████▉| 9999/10000 [37:26<00:00,  3.64it/s][2016-11-28 02:00:13,921] Making new env: MountainCar-v0\n",
      "[2016-11-28 02:00:13,926] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=10000\tepsilon=0.050\treward/step=-0.99710\tpool_size=10000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-11-28 02:00:14,664] Finished writing results. You can upload them to the scoreboard via gym.upload('/root/drqn/records')\n",
      "100%|██████████| 10000/10000 [37:27<00:00,  1.99it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = -178.700\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#the loop may take eons to finish.\n",
    "#consider interrupting early.\n",
    "for i in tqdm(range(10000)):    \n",
    "    \n",
    "    \n",
    "    #train\n",
    "    for i in range(10):\n",
    "        pool.update(SEQ_LENGTH,append=True,)\n",
    "    for i in range(10):\n",
    "        loss = train_step()\n",
    "        \n",
    "    \n",
    "    \n",
    "    if epoch_counter%100==0:\n",
    "        #average reward per game tick in current experience replay pool\n",
    "        pool_mean_reward = np.average(pool.experience_replay.rewards.get_value()[:,:-1],\n",
    "                                      weights=1+pool.experience_replay.is_alive.get_value()[:,:-1])\n",
    "        pool_size = pool.experience_replay.rewards.get_value().shape[0]\n",
    "        print(\"iter=%i\\tepsilon=%.3f\\treward/step=%.5f\\tpool_size=%i\"%(epoch_counter,\n",
    "                                                         action_layer.epsilon.get_value(),\n",
    "                                                         pool_mean_reward,\n",
    "                                                         pool_size))\n",
    "        \n",
    "\n",
    "    ##record current learning progress and show learning curves\n",
    "    if epoch_counter%500 ==0:\n",
    "        n_games = 10\n",
    "        action_layer.epsilon.set_value(0)\n",
    "        rewards[epoch_counter] = pool.evaluate( record_video=False,n_games=n_games,verbose=False)\n",
    "        print(\"Current score(mean over %i) = %.3f\"%(n_games,np.mean(rewards[epoch_counter])))\n",
    "        action_layer.epsilon.set_value(0.05)\n",
    "    \n",
    "    \n",
    "    epoch_counter  +=1\n",
    "\n",
    "    \n",
    "# Time to drink some coffee!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "iters,session_rewards=zip(*sorted(rewards.items(),key=lambda (k,v):k))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f9414452c10>]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEACAYAAACgS0HpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmcHNV1779HIwbtK9LMCG1IRkGIzWITJuBhkQyB2CKP\n+JGPDY5N3vOCHTveAG9I8RJjOw+ME2zHIcFLwI4JBowBATaDjZFAC1rBktiEZiShfQWkkea+P04X\n04xmerq7lltVfb6fz3ym53Z11ema7vrVWe654pzDMAzDMHqjj28DDMMwjGxggmEYhmGUhQmGYRiG\nURYmGIZhGEZZmGAYhmEYZWGCYRiGYZRFKMEQkctFZKWIHBKR6V2eu15E1orIcyIyq2h8uogsF5E1\nInJzmOMbhmEYyRHWw1gBXAY8XjwoIlOB9wJTgYuBW0VECk9/H7jaOTcFmCIi7wppg2EYhpEAoQTD\nObfaObcWkC5PvQf4uXPuoHPuZWAtcIaINAKDnXMLC9v9BJgdxgbDMAwjGeLKYRwNrC/6u60wdjTQ\nWjTeWhgzDMMwUk7f3jYQkUeAhuIhwAFfdM79Oi7DDMMwjHTRq2A452ZWsd82YFzR32MLYz2Nd4uI\nWKMrwzCMKnDOdU0VhCbKkFSxcfcBV4hIvYgcA7wNeNo5twnYJSJnFJLgVwH3ltqpc85+nOOGG27w\nbkNafuxc2Lmwc1H6Jy7CltXOFpH1wAzgfhF5EMA59yzw38CzwAPAx1znu7gGuA1YA6x1zj0UxgbD\nMAwjGXoNSZXCOXcPcE8Pz/0T8E/djC8GTgxzXMMwDCN5bKZ3RmhubvZtQmqwc9GJnYtO7FzEj8QZ\n7wqLiLg022cYhpFGRASX8qS3YRiGkWNMMAzDMIyyMMEwDMMwysIEwzAMwygLEwzDMAyjLEwwDMMw\njLIwwTAMwzDKwgTDMAzDKAsTDMMwDKMsTDBi5rXX4MAB31YYhmGExwQjZj7yEfjhD31bYRiGEZ5Q\n3WqN0nR0wIMPwuDBvi0xDMMIj3kYMbJkCWzdCi+84NsSwzCM8JhgxMhDD8FFF8GLL/q2xDAMIzwm\nGDEyb57mMF55BQ4d8m2NYRhGOEwwYmLXLli6FGbNglGjYP163xYZhmGEwwQjJn77Wzj7bOjfHyZP\ntjyGYRjZxwQjJubNg3e9Sx+bYBiGkQdMMGLAOU14B4IxaZIJhmEY2ccEIwZWr1bRmDpV/5482Sql\nDMPIPiYYMRB4F1JYgt1CUoZh5AETjBiYN0/nXwQEguGcP5sMwzDCIi7FVzERcWm2rztefx0aGnTu\nxbBhnePDh8PatXDUUf5sMwyjNhARnHMS9X7Nw4iYP/wBTjzxrWIBFpYy0sfLL8O11/q2wsgSJhgR\n0zUcFWCVUkbauPFG+NnPfFthZAkTjIgpnn9RjHkYRpp49VX4xS9gyxZrW2OUjwlGhKxfD5s2wamn\nHv6cldYaaeJ734MrroChQ7WjsmGUgwlGhDz8MMycCXV1hz9nHoaRFvbu1UW9PvMZaGqCjRt9W2Rk\nBROMCCme3d0VEwwjLfzoR3D++fqZNMEwKsEEIyIOHtSGgz0JxtFHw/btWnZrGL5ob4ebboLPf17/\nNsEwKsEEIyIWLoRx4/QL2B11dTBhguUxDL/ceSdMmdKZZzPBiI758+H22/M9QdcEIyJKhaMCrLTW\n8Ilz8K1vdXoXYIIRJb/8JXzsYzB7tlaf5RETjIjoaf5FMZbHMHzywANwxBFamBFgghEdra3wgx9o\n09GTT4YHH/RtUfSYYETAtm3w7LO6YFIprLTW8EngXUhRwwgTjOhobdUowje/CXfcAR/+MHziE/nK\nW4YSDBG5XERWisghEZleNH6hiCwSkWUislBEzit6brqILBeRNSJyc5jjp4VHH4Vzz4Ujjyy9nXkY\nhi8WLND+Zn/9128dN8GIjrY2GDtWHzc3w7JlOsfltNN0ueY8ENbDWAFcBjzeZXwLcKlz7mTgb4Gf\nFj33feBq59wUYIqI9BL5Tz/lhKPABMPwx7e+pfMu+vZ963hTk042zXOiNgkOHVLhHTOmc2z4cPU0\nrr9ew4Df+Q50dPizMQpCCYZzbrVzbi0gXcaXOec2FR6vAvqJyBEi0ggMds4tLGz6E2B2GBt841zP\n7UC6cswxsG6dtWIwkmX1avjjH+FDHzr8uQEDoL4edu5M3q6Au+7SLglZZvNmGDFCz2UxIvD+92sV\n5b33qnC0tvqxMQpiz2GIyOXAEudcO3A0UHy6WgtjmWXlSujXD972tt637d9f25tn+QNjZI9vfxuu\nuUbFoTt8h6Vuugl+/3t/x4+C1tbOcFR3TJwILS1wwQVa0vzLXyZlWbT07W0DEXkEaCgeAhzwRefc\nr3t57TTgn4CZpbYrxZw5c9583NzcTHNzc7W7ioXAu5AyO88HpbUTJsRrl2EAbNgAd9+ta7H0RCAY\nxx+fnF3FtLVpWCzL9CYYoHOxvvAF9TLe9z74zW/glltgyJDwx29paaGlpSX8jnqhV8FwzlV1sReR\nscDdwJXOuZcLw23AuKLNxhbGeqRYMNLIQw9pJUS5BJVS558fn02GEfDd78KVV8LIkT1v49PD6OhQ\nUasFwQg4/XR45hn49KfhlFO0xfw73hHu+F1vpufOnRtuhz0QZUjqzXtsERkK3A9c65xbEIwX8hq7\nROQMERHgKuDeCG1IlH374KmnKrv4W+LbSIpdu+Df/x3+4R9Kb+dTMLZu1XYlr77q5/hRUYlgAAwc\nqA0gb7oJ/uqv4Ctf0fOQdsKW1c4WkfXADOB+EQmmqnwcmAx8RUSeEZElIhIsTnoNcBuwBljrnHso\njA0+aWnReOTgweW/xpdgPPecJdtrjR/+EC6+WOPnpfApGG2F+EIteRjFvOc9WnK7cCGccw48/3z0\ntkVJryGpUjjn7gHu6Wb868DXe3jNYuDEMMdNC+VWRxXjQzDWrNFlY0eMgEsugXe/G2bN0rscI5/s\n36/hqAce6H3bpiZYvDh+m7qjrU0bc9aqYAA0Nur/6V/+Bc46S5d5Pu64aO2LCpvpHYJy518UEwhG\nknXvzzyjIrFwoXpEt96qF4lLL9VW1zZxK3/87Gdw0knaoqI3fHsY06fXtmCAFs184hN6A/r009HZ\nFTUmGFXy0ktau17OF7KYESNULLZvj8eu7li2TO2cMAE+/nF45BGte3//++Gxx2DaNJgxA77xDVi1\nyiZxZZ2ODi2lLW4yWArfgnHyybBjRzZi+N3hXKenFJZJk9LdPsgEo0qCcFSfCs+gSPJhqUAwihk6\nVJfovOMOvbv7+tf19yWX6JyST39aczQHDyZnpxEN992npZrlVqD7Fozx43V+UlY7vG7dCoMG6Tyr\nsJhg5JRq8hcBSTch7E4wiqmv1wlFt9yintPdd2tbg89+FhoatCzzrrtg9+7kbDaqwzm48cbDmwyW\nYuhQvbvfty9e27ojuDNvbMxupVTYcFQxxxyj38G0YoJRBe3tGsqZWeV0xCQ9jG3bdA3n3iplAkRU\nXL78ZVi0SMXmHe+A227TL/aMGdobZ948PxcYozRPPKF3vJddVv5rRPx5GcWCkdU8RpSCYR5GDpk/\nX8M2o0dX9/okBWPZMk1+lnu32ZWxY+GjH9Xe/lu2aOvm+noNYTU0aEv3L31Jl6fNUxvnrHLjjeoZ\n1tVV9jpfgtHaqoLR0GCCAdq8cNu29H6XTDCqoJzV9UqRtGBUmpjviX79NC4+d672/tm8WR+DTjwa\nNQre+U644QbNf7zxRjTHNcpj5Uotj/3AByp/rQ/BeO01vTCOHGkeRkCwlPPLL0ezv6gxwaiCaspp\ni8mqYHRlwAC48EL42te0G+qmTdorZ/9+uO46FZDzz4evflVDJQcOxGOHoXz721qa2a9f5a/1IRht\nbXpHLWKCUUyaw1ImGBWyebNe7GfMqH4fY8dqnDkJtzNOwejKoEHqeX3zm7pgT1ubhkd274ZPfUrv\nJB/K7Lz+dLN+Pfz61xo+rAZfghFcaC3p3YkJRo54+GE47zxdG7la6uq0lDDuaoj2dl0L4YQT4j1O\nTwwZAn/xF3rnu2iRLuJzxx1+bMk7N90EH/ygVrdVgy/BCOYumIfRiQlGjggbjgpIorT2T3/SeGgU\n9eFRMGuWThq0iYHRsmMH3H57700GS+FbMLKa9HauM3EfFZMmpbe01gSjAjo61MMIk/AOSCKPkWQ4\nqhwmT1bxWrnStyX54tZbtfVLmLtc34KRVQ9j506NNlTSgLQ3jjnGPIxcsHQpDBtW/pyGUtSiYECn\nl2FEw+uvw/e+B5/7XLj9+BaMYcP0vaS1nLQnog5HQadgpNETN8GogKjCUVC7gjFzpglGlPz4x7og\nz7Rp4fZz1FFanJBkJVuxYASVUllLfMchGEOHaqVbGlulmGBUQNj5F8UES7XGhXPqEaVNMM4/X0ts\nbY5GeA4dgu98B669Nvy++vTRiahJhoW6NuwzwegkrYlvE4wy2b0blizRiWlRMGmSTs6Ja1GjTZs0\n5zJmTDz7r5bhw/Vu+MknfVuSfe6+Wy/yZ58dzf6SDEsdOqSf0eLPZxbzGCYYRrc89pjOvYhq0aEB\nA7TVeVvJFc2rJwhHVdsSJE4sLBUNgXcR1f84ScHYvFnzFvX1nWNZrJQywcgJe/dqD6V/+7do7uKj\nDEcFxFlam8b8RcDMmVptZlTP1q1aNn3ppdHtM0nB6G79CPMwOjHBSJh16/RL9bOfaVLwj3+sfl/O\nqWBElfAOiDPxnWbBmDFD1y7eutW3JdllwQI488zKmwyWwgSjcuISjLS2Oc+tYLS1wfHHw+OPa8nh\nFVfoug4bNlS+r7VrddZ02EqUrtSqYNTXw7nnaodbozrmz9e281FiglE55mHkhGD2pQj8zd/Ac8/B\nuHEaprrxRm2QVy7z5un8gajzAXEJxhtv6Idt6tTo9x0VlscIx5NPwllnRbvPpAWj64U2a1VSu3dr\nuHvo0Oj3PW6cimfaGnbmVjC6fiAHDdI1qxcs0PDUCSfAb35T3r6inH9RTFyltatWwbHHwpFHRr/v\nqJg1S/MYaZyclHYOHtTeXGeeGe1+fXsYWUt6B9eYOApL+vbVfa9bF/2+w5Brweiuv8vb3qZrHt9y\ni65bfemlGnLqif37de2HCy+M3sa4PIw0h6MC/uzPVCzWrPFtSfZYvly7DQwbFu1+0yIYWbmJiLqH\nVFfSGJbKrWD09s+8+GJYsULnVZx1lq7fsGfP4ds98YTmLkaMiN7Go45Sl3bHjmj3u2wZnHJKtPuM\nGhELS1VLHOEo0Av2li3xzQ0qpjvBGDRIk/jdfQ/TSFz5iwATjATpLkbalfp6TYivWKF3Vscdp1VV\nxXc48+ZFX04bIBKPl5EFDwM6w1JGZTz5ZPQJb9AmesOHJ9OSoqcIQJYS3yYYOaISd7GpSXvy3HUX\n3Hwz/Pmf66xuiGf+RTFRC4Zz2RGMCy7QKrb2dt+WZIv58+PxMCCZsNTevfo/7y6klqXEd9yCkcbS\n2lwKxv79sGuXtk2ohLPOgqefhg99SBf+ueoq/VCcfno8dkL0grF+vbYQHzUqun3GxahR+v6fesq3\nJdlhwwYN2UyZEs/+kxCMwLvoLllsHkYn5mEkxIYN+sHvU8W769MHrr5aZ9GOGKGP+/aN3saAqCul\nsuJdBNis78oIvIu4Wr4kKRjdkaVKqSQE44UX0lUEkEvBiKJ6YdgwDU99+9vR2NQTUXsYWRMMWx+j\nMuIMR0EyglHq+2keRifDh+uNQdRFMWHIpWCUk/BOC7UuGGefrSvw7dzp25JsEFfCO8C3h5EVwXjt\nNdi3Tysd40IkfWGp3ApGnPXRUTJunHbujGp9iDSugVGKfv30Avi73/m2JP3s3683BHHm1EwwyqNU\nHiZKTDASIO4JNVHSty+MH69rY4Rl717N38SVEI0LC0uVx5IlWvodVYv97kiDYGShSirucFSACUYC\nZCkkBdGFpVas0P5RcSbp48Am8JVH3OEo8C8YWUl6JyUYaSutzaVgZMnDgOgEI2v5i4ATT1TvKE13\nUmkkKcGIuz1HqRu60aM1RNvREd/xo8A8jByRpRwGRFdam1XBsDYhveNcfC1BiunfX/NKcVXmHDyo\nM8kbG7t//sgjYfBg2L49nuNHhQlGFYjI5SKyUkQOicj0bp4fLyJ7ROTTRWPTRWS5iKwRkZvDHL87\nOjrUpU7bWtalqHUPA0wwemPdOhXWCRPiP1acYalXX4WRI7UNSU9kIfGdlGBMmKDHOngw/mOVQ1gP\nYwVwGfB4D8//M/BAl7HvA1c756YAU0Qk0sYbmzdrf/p+/aLca7xEIRgdHZrDOOmkaGxKmgsv1Eqp\nJBrfZZEgHJXEGu1xCkY53r8JRif19Xo+1q+P/1jlEEownHOrnXNrgcM+xiLyHuBFYFXRWCMw2Dm3\nsDD0E2B2GBu6krWEN6jb+fLL4eK2L76oM9OHD4/MrEQZM0YvJIsW+bYkncQ9Ya+YNAhG2iulkhIM\nSFdYKpYchogMBD4PzOWtYnI00Fr0d2thLDKylr8ALZMcNqy65WMDshyOCrCwVM8kkfAO8C0Yaa+U\n2r9fczyV9qqrlkwJhog8Usg5BD8rCr//ssTL5gA3Oedei8zSMslahVRA2LCUCUZ+2bcPVq+G6Ydl\nCeMhTsEo5/uZ9pBU0Kuuri6Z46VJMHqt2HfOzaxiv2cC/0tEvgUMBw6JyBvA3cC4ou3GAm2ldjRn\nzpw3Hzc3N9Pc3FzywFkMSUGnYLzzndW9ftkyuPLKaG1KmnPPhfe+V7uxDh7s25r0sHCh5qaSWnK3\nqUmPGQdtbdrWvhSNjZqPSytJhqNA52Lcd1/pbVpaWmhpaYndliineL0ZenLOnfvmoMgNwB7n3K2F\nv3eJyBnAQuAq4JZSOy0WjHJoba3+ouuTsKW1y5bBd74TnT0+GDhQ2148/rgunWsoSYajwH9IKu0e\nRtKCUY6H0fVmeu7cubHYErasdraIrAdmAPeLyINlvOwa4DZgDbDWOfdQGBu6ksUcBoQLSe3cCdu2\n6T6yjoWlDscEI12kUTCSIpSH4Zy7B7inl23mdvl7MXBimOOWIsshqWo/FMuX62zpatb/SBszZ+rC\nVYbinFZI/ehHyR0zLsFwLh9VUq2tMHFicscbNUqbk+7apVMGfJKDS8xbqcWkdx4S3gFvf7vOpWlt\n7X3bWmDNGhgyRC/iSTFkiM6H2bs32v3u3t25/1KMHKlVSGldujdpDyNoc56GnlK5Eozdu/UuxrcK\nV8OoUXDgQHXrQuRJMOrqNClqYSkl6XAU6AUqDi8j8P57m3xYV6frTGzZEu3xoyJpwYD0hKVyJRhJ\n9aiPA5HqvYysrYHRG5bH6CTJCXvFxCUY5Xr/ac5jmGDkhKyGowKqqZQ6eBCefVZzGHlh5kx49NH0\ndyxNAh8eBphg9ER7e+nmiXGRljbnuRKMrCa8A6rxMNas0S/hoEHx2OSDCRO0xcmyZb4t8cvOndp0\n0Ed/sDQIRhoT35s2afi4VPPEODAPIwayWlIbUI1g5Cl/UYyFpeCpp+C00/wsiOVbMNLaHsRHOApM\nMGIh6yGpakpr8yoYs2bBww/7tsIvvsJR4F8w0hqS8iUYEyeqt+m7m3OuBKMWQ1J5FYzmZr3Dfv11\n35b4I4kFk3oiDsGo5IbOBOOt9O+v5cZhGpRGQa4EI+sexvjx+iXZv7/81+RVMIYM0ff1hz/4tsQP\nhw7B00/DjBl+jm8eRvf4EgxIR1gqV4KRdQ+jb18YN07XxiiHLVv0Dnz8+FjN8kYth6VWrdKL9lFH\n+Tl+1ILR3q7taxoaytveBONwTDAi5MCBZHvUx0UlpbXLlmkFTRbnnZRDLSe+fYajQMMfe/ZU5u2W\nYuNG/W6Wm8BPa5WUT8E45hgTjMjYsEHvXpLqUR8XleQx8hqOCjj9dHjllXReOOJm/nx/CW/QvmRR\nVipVWsE4dKj2T0pbDsu3h+F7LkZuBCPr4agAE4xO+vaF887TSXy1hs8KqYAow1KVCoaIClaabhYO\nHdLzMWaMn+NbSCpCsj4HI6CS0tq8CwZoWKrW8hibN2t+aupUv3b4FAxIXx5j82adUJrUQlZdMcGI\nkKxXSAWU62EcOKCzvKdNi98mnwR5DOd8W5IcCxZodZTvdvVRC0alEYC0CYbPcBTo+di1S5fs9UVu\nBCMvIakgTtlbH6XnntMkWP/+ydjli8mT9Y7u2Wd9W5IcaQhHgXkYXfEtGH36+O8plRvByIuHMWiQ\nrmfd2xe1FsJRoLHsWiuv9V0hFeBbMNKWw/AtGOA/LJUbwciLhwHlhaWWLoVTTknGHt/UUnltezss\nWQJnnunbEv+CYR7G4ZhgRERekt5QnmDUiocBcP758MQT0c0JSDNLl+r/v7dV6ZIgKsFwrroIgAnG\n4VhIKgI6OnQehq9yt6jprVLKudoSjBEjtGLoySd9WxI/vhZM6o6oBGPHDqivr7wFvwnG4ZiHEQFb\nt2rcPy8J4N48jA0bNAGW9CIuPqmVsFRaEt6gOYStW8N3SK3W+zfBOBwTjAjIUzgKeheMwLvIa0uQ\n7jDBSJ6+fdW727w53H6q/X4GSe80lFQ7l47rTBCS8nVOciEYeamQCihXMGqJs87SeSfbtvm2JD5a\nW7UdxuTJvi3pJIqwVLUX2oEDtdXPnj3hjh8FW7eqPQMG+LUjqKL05XnlQjDyVCEF2qTtjTd0kk53\n1KJg1NfDOefAb3/r25L4CPIXafIcfQoGpCcslYZwVIDPsFQuBCNvHoZI6a61tSgYkP+wVJrCUQEm\nGIoJhpILwcibhwE9h6Vef12XajzuuORt8s2sWfluE5KmCqkAEwwlTYLhs825h+XloycNyaio6am0\nduVKmDJFQzS1xnHHafO3UaP0HBx7rP4OHh97rMaZs8jrr8OKFXDaab4teStNTeHbsoS5oTPBOJxJ\nk+D3v/dz7FwIRt5CUqCC8cwzh4/XajgKNFS3ZIlWzqxdq0nwtWvhzjv19/PPa1VPsZgEvydN8tdl\ntBwWL9ZGkr6Tql1pagqfNwpzQ5eW9iCtrbrOfBqYNAluv93PsXMhGHkNSd111+HjtSwYoKLR2Kg/\n55zz1uc6OvSLHQjJmjXw2GP6eN06vWgdeyxccgn8/d/7sb8n0hiOgvAhqf37YefO6lfCbGzUc+Ob\ntHkYFpKqkj17tP/OsGG+LYmWnpLey5bB7NnJ25MF+vTR9c3Hj4cLL3zrc+3tulb6qlXw/vfDxz/u\nv314MU8+CVdc4duKwwkrGBs36kW/2nNtIanDOfpoLfN94w3o1y/ZY6foK1MdgbubplLEKJgwQb9s\nBw50jjkHy5fXtodRLUccod7F7NmaB1m3zrdFnTiXzgopUMHYtKn6QoOw4eI0CEbQCystglFXpzdF\nL7+c/LFzIRhp+UdGyRFH6Bet+EOxbp0mdY86yptZueD449O1vsZLL+n/e9w435YcTr9+mlfZvr26\n14ctSEmDYOzcqbPeBw/2a0cxvsJSmReMPCa8A7qW1tZ6/iIqpk3T0FRaSKt3ERAmLBVWMEaP1uVq\ne1tQLE7S5F0E+Cqtzbxg5NXDgMNLa5cuNcGIgrR5GCYYPVNfr3f21Xo4UZBGwQhW5kyaXAiGeRhG\nJaTNw0hrhVSAT8EA/2GptAqGeRhVYCEpo1KmTtU10X2GOQL27NGy37e/3bclPWOCYYIREEowRORy\nEVkpIodEZHqX504SkScLzy8TkfrC+HQRWS4ia0Tk5jDHh3yHpIpLa3fv1i/NlCl+bcoDw4bpz/r1\nvi2Bp5/WpXbTPHPfBCN915hAMJJukxPWw1gBXAY8XjwoInXAT4H/65w7AWgG2gtPfx+42jk3BZgi\nIu8KY0DeQ1LBh2LFCg2l1NX5tiofHH98OsJS8+enO38B1QuGc7rYlwlG9AwdqjcZW7cme9xQguGc\nW+2cWwt0nQUxC1jmnFtZ2G6Hc86JSCMw2Dm3sLDdT4Cqp6G1t+sJa2iodg/pZvBg7X+/caOFo6Jm\n2rR0JL7TnvCG6gVj2zZdBTNsu5PGRr/tQdIoGOAnLBVXDmMKgIg8JCKLRORzhfGjgdai7VoLY1Wx\ncaOKRd/Mz1fvmSCPYYIRLWnwMDo6YMGCdCe8oXrBiCpc3NBgHkZ3+Cit7VUwROSRQs4h+FlR+P2X\nJV7WFzgb+BvgHOAyETkvIpvfJM/hqIAgLGWCES1p8DBWr9ZZ52n3kMMIRhTfT58hqT174ODBdLYe\n8lFa2+u9uXNuZhX7bQV+75zbASAiDwDTgf8CiuezjgXaSu1ozpw5bz5ubm6muahlZFqVP0omT9Yq\nmpUr4aSTfFuTH4K5GM75ayuThXAUaGjUOb14VjLbOaoKRp+CEXhJaWw9NGmSFk0AtLS00NLSEvsx\nowzmFJ/SecDnRKQfcBB4J/DPzrlNIrJLRM4AFgJXAbeU2mmxYHSlFjyMSZPgX/9V14AYOtS3Nflh\n2DAYMkQrpcaP92PDk0+mPxwFerEMvIxKBCMPHkaab0onTYKf/1wfd72Znjt3bizHDFtWO1tE1gMz\ngPtF5EEA59xO4P8Bi4AlwCLn3EOFl10D3AasAdYWjVdMnudgBEyeDAsXWjgqDnznMbJQIRVQTVgq\nKsEYOVL7ObW3975t1KRdMFKXwyiFc+4e59w451x/51yTc+7ioufucM6d4Jw7yTl3fdH4Yufcic65\nY51znwxz/DzPwQiYPFl/m2BEj888xvbtejE64QQ/x68Un4JRV6cNN7dsCb+vSkmzYIwbd3hH67jJ\n9EzvWghJNTZqWaIJRvT49DAWLIDTT89OhZ9PwQB/Yak0C0bQ0fqVV5I7ZqYFoxZCUiJw5ZXZCV1k\nCZ9NCLMUjgITjLSSdFgqs4IR1SzSLPCDH+gXxoiW4kqppFmwAGbMSP641VKpYLz+OuzdG93aLSYY\n3ZP0XIzMCsa2bRqqCTuL1KhdRozQBalaW3vfNkqcg8WL4bTTkj1uGCoVjA0b9DVRLYNrgtE9Sc/F\nyKxgpP0faWQDH63OX3pJW76kfcJeMZUKRtT5RR+CEbWXFAcWkiqTWkh4G/HjI4+xaBGcemqyxwyL\nb8FoaEieNg5eAAARdElEQVS+n1TwHtI4aS/ABKNMaiHhbcSPDw9j0aJshaNA50Ls2wdvvFHe9lGX\nvPvwMLIQxTDBKJNamINhxI8vDyNrgiFSWRPAPISksiAYI0ZoE8sdO5I5XqYFwzwMIyxJV0p1dGjC\nO2shKagsLBV1BMAEo3tEkvUyMisYFpIyomDkSF2zoa1kC8zoeP55vStMcyK1JyoRjKhv6IYOhf37\nNRGdFFkQDEi2tDazgmEhKSMqksxjZDEcFeBTMIKQWJKJ76wIRpKltZkWDPMwjChIMo9RC4LR0aHb\njRkT7fGTDktlSTDMwyhBUK0xYoRvS4w8YB5GeZQrGFu2aOv4fv2iPb4JRveYYPRCFuqjjeyQlIdx\n6BAsXQrTp8d/rDgoVzDi8v6TFIwDB7Sj8OjRyRwvDCYYvWAJbyNKAg8j7kqpNWv0AjR8eLzHiYta\nEoygtUldXTLHC8OECboQ2MGD8R8rk4JhCW8jSkaO1PDJhg3xHifL4SioLcHISjgK4MgjtSAgiZ5o\nmRUM8zCMKEliMaWsC8bo0dr0s7c72bi+n0lWSWVJMCC50tpMCoaFpIyoSWIxpawLRt++6o1t3lx6\nO/MwkiepPEYmBcNCUkbUxO1hHDwIy5ZlN+EdUE5YygQjeZKai5FZwTAPw4iSuD2M557TC9CQIfEd\nIwnKEYy4LrZBL6sk2rhkUTDMw+iBrP0zjfQTeBhxXYyyHo4K8OlhDByo61jv3h39vruStWuMCUYP\ntLfD1q22ZKkRLUcdpRejStetLpdaEYx9+7TnU1yTaivpmBsGE4zuyZxgbNqkX+6+fX1bYuSNOCfw\n1YpgtLVpS5C4JtU2NsZfKXXwoCb2s3RTOno0vPZa/N5X5gTDEt5GXMTVIqS9HVauhFNOiX7fSVOO\nYMSZX0wi8b1pE4wapR5nVgjanMed+M6kYFjC24iDuDyMVatg4kRdxzvr1IJgZC0cFZDEXIzMCYbN\nwTDiIi4PIy/hKDDBSDPmYXSDhaSMuIhr9b08CUaQQ+jpHJlg+COJxHcmBcM8DCMORo/WZnNRX5Dy\nJBj9+ml567Zt3T8f9/cziSopE4yeyZxgZPWfaWSDqPMY+/fr/k4+Obp9+qZUWCoJDyPuKqmsXmNM\nMLrBPAwjTqLOY6xYAcceCwMGRLdP3/gWDPMwumfiRHj5ZV3xMC4yJRjOmWAY8RK1h5GncFRAT4Jx\n6JDe/Ue9NGsxo0frin5xXhSzKhgDBuiEyTjb9GdKMLZv74yhGkYcRO1h1JJgvPqqLg5VXx/fsevr\ntR9XTzmUsMS1HnlSxF1amynBMO/CiJugCWFUlVKLF8Opp0azr7TQk2Ak9f2MM/G9eTMMHaqLEmWR\nuPMYmRIMm4NhxM3o0dCnTzSJ1ddfh9Wr4aSTwu8rTfgWjDgT31kNRwXEPRcjU4JhczCMuBGJLo+x\nfDkcd5yGUfNEGgQjLg8jD4JhHkYBC0kZSRBVHiOP+QswwUgzqRYMEblcRFaKyCERmV40fqSI3CEi\ny0VklYhcV/Tc9ML4GhG5uZLjZf2faWSDqDyMvAtG1zyPCYZ/Ui0YwArgMuDxLuNXADjnTgJOAz4s\nIuMLz30fuNo5NwWYIiLvKvdg5mEYSWAeRmkGD9bQ3Z49bx03wfBPUxNMnRrf/kMJhnNutXNuLdC1\n+/0mYKCI1AEDgP3AbhFpBAY75xYWtvsJMLvc45lgGEkQRaXUvn3wwgtwwgnR2ZUmugtL5aFKKuuC\n0acP/O53Me4/jp065+YBu4GNwMvAd5xzO4GjgdaiTVsLY2WR9X+mkQ0aGvT35s3V72PpUhWLOOck\n+MSnYFiVlD96XbdORB4BGoqHAAd80Tn36x5e8z6gP9AIjAT+ICKPVmPgnDlzAF2EZu/eZkaObK5m\nN4ZRNsWVUg0NvW/fHXkNRwV0FYw9e3SlumHD4j92XCEp57Jbut/S0kJLS0vsx+lVMJxzM6vY79nA\nr5xzHcAWEfkjmst4AhhXtN1YoK3UjgLBWLsW7rwzvqUfDaOYadNUMM47r7rXL1oEzc2RmpQqugpG\n4F0k8f0cORJ27tSbyChXxdu2TdtrZLGTRHNzM81FH7i5c+fGcpwoQ1LFH5U/ARcAiMhAYAbwnHNu\nE7BLRM4QEQGuAu4tZ+c2B8NIkiCPUS2LF9eWh5HknXldnS6hGiZk2B0WjuqdsGW1s0VkPSoI94vI\ng4WnfgjUi8gK4CngNudc8PW7BrgNWAOsdc49VM6xLOFtJEmY0to9e2DdOt1HXunJw0iKOMJSJhi9\n02tIqhTOuXuAe7oZ3w+8v4fXLAZOrPRY9s80kiRMae0zz2g7kCjDJWmjO8FI8vsZR6WUXWN6JzMz\nvc3DMJKksVGTuFu2VP7avCe8IR0eRtSVUiYYvWOCYRjdIFK9l2GCET8WkvJDZgTD/plG0lSbx6gF\nwRgxAl57TTvygglGrZAZwTAPw0iaajyMnTv1zvu44+KxKS2IvPWibYJRG2RCMA4e1HhlU5NvS4xa\nohoPY8kSOOUULf3MO0FYKsj1NDYmd+yok97BpD0TjNJkQjBefRWOOirfVSdG+qjGw6iFcFTAmDEq\nGJs2Jf/9jDrpvWuX9mEaMiS6feaRTAiGhaMMHzQ1wYEDlVVKLVqUvyVZeyLwMHx8P6MOSa1fb95F\nOWRCMMxVNHwQVEpVEpbK+wzvYnwKxtChsH+/Jt6j4Mc/hvPPj2ZfeSYTgmEehuGLSvIY27erNzJl\nSrw2pQWfghEk3aMIS23cCP/xH3D99eH3lXdMMAyjBJXkMRYvhunTNRZeCwSC4avDa1RhqW9+E666\nyq4x5ZCJj7aFpAxfVOJh1FLCG/x6GBBNpVRrK/z0p3Dddb1va2REMMzDMHxRiYdRy4Lh44YuipDU\nN74BV1+dbElwlsmEYGR1URMj+4wZo8nVrVt737bWBGPUKNixQzvzZjEktW4d/OIX8PnPR2dT3km9\nYDhnHobhj+LV90qxeTPs3g2TJydjVxqoq9P5Fy+9lE3B+NrX4CMfUeEzyiP1grFzp04IGjzYtyVG\nrVKOYCxerPMvam1FyKYmGDTIz4S3MILxwgvwq1/BZz4TrU15J9R6GElgCW/DN+XkMWppwl4xTU2w\nb5+fY4dJen/1q/Dxj2sTRaN8Ui8YFo4yfHP88XD//aW3WbwY3ve+ZOxJE01NnR1rk6ZaD2P1avjN\nb+D556O3Ke+kPiRlgmH4plwPo5YS3gFNTf6+nw0NWiXlXGWvmzsXPvUpnS1uVEbqPQwLSRm+Ofpo\nbUGxbRuMHHn48xs36l32xImJm+adSy/VGe4+GDhQ85u7d5d/8V+1Ch59FH74w3htyyvmYRhGL/RW\nKRX0j6q1hDfAGWfARRf5O36lYak5c+Czn7UimmoxwTCMMijVhLBWw1FpoBLBWLYMnngCrrkmXpvy\nTOoFw0JSRhoo5WGYYPijkkqpG26Aa6/VUJZRHakXDPMwjDTQU+LbORMMn5TbHmTRIli4ED784fht\nyjOpF4w9e3Q2qWH4pCcPo61NRcO8YD+UG5K64Qb4whegf//4bcozqReMMWNqp120kV7GjYO9e7V3\nUjHBhL1aTHingXIEY/58WLEC/u7vkrEpz6T+UmzhKCMNiMDUqYd7GRaO8ks5gvGVr8CXvgRHHpmM\nTXkm9YJhrr6RFrrLY9TSkqxppLek9+9/r32jPvjB5GzKM6kXDPMwjLTQNY9hCW//lPIwnFPv4stf\n1gl+RnhMMAyjTLp6GK+8oheiMWP82VTrjB6ta5V0dBz+3GOPwYYNcOWVyduVV1IvGBaSMtJCVw/D\nvAv/1Ndra/Vt29467px6FjfcAH1T3wApO6ReMMzDMNLC+PHat2jnTv3bBCMddBeWmjdPK9quuMKP\nTXkl9YJhHoaRFrpWSplgpIOughHkLubM0VUBjehIvWA0Nfm2wDA6CfIYQcK7FhdNShtdK6Xuvx/e\neAMuv9yfTXkl9YJRX+/bAsPoJMhjvPiiLk3a0ODbIqPYwwi8i7lzbcJvHNgpNYwKCDwMC0elh+J+\nUr/6lYYOZ8/2a1NeCSUYIvItEXlORJaKyP+IyJCi564XkbWF52cVjU8XkeUiskZEbg5zfMNImsDD\nsAl76SHwMDo6tCrqH//RWrXERVgP42FgmnPuFGAtcD2AiBwPvBeYClwM3Cry5r/w+8DVzrkpwBQR\neVdIG2qClpYW3yakBp/nYvx4rZJ69NF0CIZ9LjoFY86cFgYOhEsu8W1RfgklGM65R51zwZSZBUBQ\n0/Ru4OfOuYPOuZdRMTlDRBqBwc65hYXtfgKY81gGdmHoxOe56NNHK6WeeSYdCW/7XGgeqa0Nbr21\nxbyLmIkyh/Eh4IHC46OB9UXPtRXGjgZai8ZbC2OGkRmmTdP1u63tfjpobIQ//QkGDICZM31bk296\nnQMpIo8AxbUgAjjgi865Xxe2+SLQ7py7MxYrDSNFTJsG+/b5tsIIGDlSZ3Ofd555F3EjzrlwOxD5\nW+D/AOc75/YXxq4DnHPuxsLfDwE3AOuAx5xzUwvjVwDvdM59tId9hzPOMAyjRnHORS6fobqsiMhF\nwOeAcwOxKHAf8F8ichMacnob8LRzzonILhE5A1gIXAXc0tP+43jDhmEYRnWE8jBEZC1QDwStvxY4\n5z5WeO564GqgHfikc+7hwvipwO1AP+AB59wnqzbAMAzDSIzQISnDMAyjNkjlTG8RuUhE/lSY3Het\nb3viQETGisjvRGSViKwQkb8vjA8XkYdFZLWIzBORoUWvye1kSBHpIyJLROS+wt81eR4ARGSoiPyy\n8P5WiciZtXg+Cu9rVeE9/JeI1NfSeRCR20TkVRFZXjQW2fsvnM+fF14zX0TG92qUcy5VP6iIPQ9M\nAI4AlgLH+bYrhvfZCJxSeDwIWA0cB9wIfL4wfi3wzcLj44Fn0LzTxMI5CjzEp4DTC48fAN7l+/1V\ncT7+AfgZcF/h75o8DwXbbwc+WHjcFxhaa+ej8P1/Eagv/P0L4AO1dB6APwdOAZYXjUX2/oGPArcW\nHv9vdO5cSZvS6GGcAax1zq1zzrUDPwfe49mmyHHObXLOLS083gs8h058fA/w48JmP6ZzYmNuJ0OK\nyFjgL4B/LxquufMAUGivc45z7j8BCu9zF7V3PnYDB4CBItIX6I/O56qZ8+CcewLY0WU4yvdfvK+7\ngAt6symNgtF10l/uJ/eJyET0TmIB0OCcexVUVIDRhc3yPBnyJrTarjihVovnAeAYYKuI/GchRPdv\nIjKAGjsfzrkdwD8Dr6DvaZdz7lFq7Dx0w+gI3/+br3HOHQJ2isiIUgdPo2DUFCIyCFX3TxY8ja5V\nCLmuShCRS4BXC95WqTLqXJ+HIvoC04F/dc5NB/YB11F7n4tJaJhyAjAG9TTeR42dhzKI8v33Oo0h\njYLRBhQnX8YWxnJHwdW+C/ipc+7ewvCrItJQeL4R2FwYbwPGFb08OC89jWeFs4F3i8iLwJ3A+SLy\nU2BTjZ2HgFZgvXNuUeHv/0EFpNY+F6cBf3TObS/c/f4KeAe1dx66EuX7f/M5EakDhjjntpc6eBoF\nYyHwNhGZICL1wBXoRMA88h/As8657xaN3Qf8beHxB4B7i8avKFQ2HEPnZMhNwC4ROUNEBJ0MeS8Z\nwTn3BefceOfcJPR//Tvn3JXAr6mh8xBQCDesF5EphaELgFXU2OcCLQKZISL9CvZfADxL7Z0H4a13\n/lG+//sK+wD4a+B3vVrjuxKgh+qAi9APzFrgOt/2xPQezwYOoVVgzwBLCu97BPBo4f0/DAwres31\naPXDc8CsovFTgRWF8/Vd3+8txDl5J51VUrV8Hk5Gb5yWAnejVVI1dz7QvNYqYDmanD2ils4DcAew\nAdiP5nI+CAyP6v0DRwL/XRhfAEzszSabuGcYhmGURRpDUoZhGEYKMcEwDMMwysIEwzAMwygLEwzD\nMAyjLEwwDMMwjLIwwTAMwzDKwgTDMAzDKAsTDMMwDKMs/j8Ozzt93lGNTgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f94144ec9d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(iters,map(np.mean,session_rewards))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}