mountaincar-neural_network-Copy3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Whatami\n",
    "\n",
    "I am a simple experiment on using qlearning agent setup for MountainCar problem.\n",
    "Being off-policy value based algorithm, qlearning has comparatively poor convergence on this problem (see a2c nearby for comparison) yet it does manage to find some policy that brings him to the end.\n",
    "\n",
    "\n",
    "## About OpenAI Gym\n",
    "\n",
    "* Its a recently published platform that basicly allows you to train agents in a wide variety of environments with near-identical interface.\n",
    "* This is twice as awesome since now we don't need to write a new wrapper for every game\n",
    "* Go check it out!\n",
    "  * Blog post - https://openai.com/blog/openai-gym-beta/\n",
    "  * Github - https://github.com/openai/gym\n",
    "\n",
    "\n",
    "## New to Lasagne and AgentNet?\n",
    "* We only require surface level knowledge of theano and lasagne, so you can just learn them as you go.\n",
    "* Alternatively, you can find Lasagne tutorials here:\n",
    " * Official mnist example: http://lasagne.readthedocs.io/en/latest/user/tutorial.html\n",
    " * From scratch: https://github.com/ddtm/dl-course/tree/master/Seminar4\n",
    " * From theano: https://github.com/craffel/Lasagne-tutorial/blob/master/examples/tutorial.ipynb\n",
    "* This is pretty much the basic tutorial for AgentNet, so it's okay not to know it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting https://github.com/yandexdataschool/agentnet/archive/master.zip\n",
      "  Downloading https://github.com/yandexdataschool/agentnet/archive/master.zip\n",
      "\u001b[K     \\ 4.6MB 62.7MB/s\n",
      "Requirement already up-to-date: six in /Users/alexajax/anaconda3/lib/python3.5/site-packages (from agentnet==0.10.1)\n",
      "Requirement already up-to-date: lasagne in /Users/alexajax/anaconda3/lib/python3.5/site-packages (from agentnet==0.10.1)\n",
      "Requirement already up-to-date: theano>=0.8.2 in /Users/alexajax/anaconda3/lib/python3.5/site-packages (from agentnet==0.10.1)\n",
      "Collecting numpy>=1.9 (from agentnet==0.10.1)\n",
      "  Downloading numpy-1.11.3-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (4.2MB)\n",
      "\u001b[K    100% |################################| 4.2MB 105kB/s \n",
      "\u001b[?25hCollecting scipy>=0.11 (from theano>=0.8.2->agentnet==0.10.1)\n",
      "  Downloading scipy-0.18.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (21.0MB)\n",
      "\u001b[K    100% |################################| 21.0MB 20kB/s \n",
      "\u001b[?25hInstalling collected packages: numpy, agentnet, scipy\n",
      "  Found existing installation: numpy 1.11.1\n",
      "    Uninstalling numpy-1.11.1:\n",
      "      Successfully uninstalled numpy-1.11.1\n",
      "  Running setup.py install for agentnet ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \bdone\n",
      "\u001b[?25h  Found existing installation: scipy 0.18.0\n",
      "    Uninstalling scipy-0.18.0:\n",
      "      Successfully uninstalled scipy-0.18.0\n",
      "Successfully installed agentnet-0.10.1 numpy-1.11.3 scipy-0.18.1\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade https://github.com/yandexdataschool/agentnet/archive/master.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiment setup : http://192.168.0.115:9999\n",
    "* Here we basically just load the game and check that it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: THEANO_FLAGS=\"floatX=float32\"\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function \n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "%env THEANO_FLAGS=\"floatX=float32\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#global params.\n",
    "GAME = \"CartPole-v0\"\n",
    "\n",
    "#number of parallel agents and batch sequence length (frames)\n",
    "N_AGENTS = 1\n",
    "SEQ_LENGTH = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:26:47,781] Making new env: CartPole-v0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.02258126 -0.17717613  0.03649135  0.31959103]\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "env = gym.make(GAME)\n",
    "env.reset()\n",
    "obs = env.step(0)[0]\n",
    "action_names = np.array([\"left\",'stop',\"right\"]) #i guess so... i may be wrong\n",
    "state_size = len(obs)\n",
    "print(obs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x114697a20>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEACAYAAAC08h1NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFAdJREFUeJzt3X+sXOV95/H3B1wohNRxkwCJTQxZWGKiUJNszK7YKlOS\nutBKgFZaxCZdJY0iIlE22XS1i539wzfbVcErJatUKlJoftRpSYlLk8ZULT+87lQJaoAE27hch7jt\nmmA3vmQhC0vYjWz83T/m2JlcX7hzfWd8r+e8X9LIZ75zfjyP7vVnnvvMOWdSVUiS2uOUhW6AJOnE\nMvglqWUMfklqGYNfklrG4JekljH4JallRhb8Sa5K8p0k301yy6iOI0mam4ziPP4kpwDfBd4N/CPw\nCHBDVX1n6AeTJM3JqEb8a4A9VfVkVR0E7gKuHdGxJElzMKrgXw481fd8X1OTJC0wP9yVpJZZMqL9\n7gfe1Pd8RVM7Kok3CZKk41BVmc/2oxrxPwJcmGRlktOAG4At01fasGEDVTWWj3Hum/07+R/27+R9\nDMNIRvxV9VKSm4H76b25fK6qdo/iWJKkuRnVVA9VdS9w8aj2L0k6Pgv64W6n01nIw4/UOPcN7N/J\nzv6120gu4BrowEkt1LEl6WSVhFqkH+5KkhYpg1+SWsbgl6SWMfglqWUMfklqGYNfklrG4JekljH4\nJallDH5JahmDX5JaxuCXpJYx+CWpZQx+SWoZg1+SWsbgl6SWMfglqWUMfklqmXl9526SvcBzwGHg\nYFWtSbIM+DKwEtgLXF9Vz82znZKkIZnviP8w0Kmqy6pqTVNbB2ytqouBbcD6eR5DkjRE8w3+zLCP\na4FNzfIm4Lp5HkOSNETzDf4CHkjySJIPNbVzqmoKoKoOAGfP8xiSpCGa1xw/cEVVfT/J64H7kzxB\n782g3/TnkqQFNK/gr6rvN//+IMmfAWuAqSTnVNVUknOBp19u+4mJiaPLnU6HTqczn+ZI0tjpdrt0\nu92h7jNVxzcgT3ImcEpVvZDkVcD9wCeAdwPPVtXGJLcAy6pq3Qzb1/EeW5LaKglVlXntYx7BfwHw\nVXpTOUuAO6vqtiQ/D2wGzgOepHc65/+eYXuDX5LmaEGDf74Mfkmau2EEv1fuSlLLGPyS1DIGvyS1\njMEvSS1j8EtSyxj8ktQyBr8ktYzBL0ktY/BLUssY/JLUMga/JLWMwS9JLWPwS1LLGPyS1DIGvyS1\njMEvSS1j8EtSyxj8ktQyBr8ktcyswZ/kc0mmkjzWV1uW5P4kTyS5L8nSvtfWJ9mTZHeStaNquCTp\n+Awy4v8C8CvTauuArVV1MbANWA+Q5BLgemAVcDVwe5J5fSmwJGm4Zg3+qvoG8MNp5WuBTc3yJuC6\nZvka4K6qOlRVe4E9wJrhNFWSNAzHO8d/dlVNAVTVAeDspr4ceKpvvf1NTZK0SAzrw90a0n4kSSO2\n5Di3m0pyTlVNJTkXeLqp7wfO61tvRVOb0cTExNHlTqdDp9M5zuZI0njqdrt0u92h7jNVsw/Wk5wP\n3FNVb2uebwSeraqNSW4BllXVuubD3TuBy+lN8TwAXFQzHCTJTGVJ0itIQlXN66SZWUf8Sb4EdIDX\nJvkesAG4DfiTJB8EnqR3Jg9VNZlkMzAJHARuMt0laXEZaMQ/kgM74pekORvGiN8rdyWpZQx+SWoZ\ng1+SWsbgl6SWMfglqWUMfklqGYNfklrG4JekljH4JallDH5JahmDX5JaxuCXpJYx+CWpZQx+SWoZ\ng1+SWsbgl6SWMfglqWUMfklqGYNfklpm1uBP8rkkU0ke66ttSLIvyaPN46q+19Yn2ZNkd5K1o2q4\nJOn4DDLi/wLwKzPUP1VVb28e9wIkWQVcD6wCrgZuTzKvLwWWJA3XrMFfVd8AfjjDSzMF+rXAXVV1\nqKr2AnuANfNqoSRpqOYzx39zkh1JPptkaVNbDjzVt87+piZJWiSON/hvB95cVauBA8Anh9ckSdIo\nLTmejarqB31Pfx+4p1neD5zX99qKpjajiYmJo8udTodOp3M8zZGksdXtdul2u0PdZ6pq9pWS84F7\nquptzfNzq+pAs/wx4J1V9d4klwB3ApfTm+J5ALioZjhIkpnKkqRXkISqmtdJM7OO+JN8CegAr03y\nPWAD8EtJVgOHgb3AhwGqajLJZmASOAjcZLpL0uIy0Ih/JAd2xC9JczaMEb9X7kpSyxj8ktQyBr8k\ntYzBL0ktY/BLUssY/JLUMga/JLWMwS+9jMm7f5vJu397oZshDZ0XcEkv49t3fPiY2s+86jVc+r6N\nC9AaqccLuCRJc2bwS1LLGPyS1DIGvyS1jMEvSS1j8EtSyxj8ktQyBr8ktYzBL0ktY/BLM3jxmadm\nrL/+kned4JZIwzdr8CdZkWRbkseT7Erykaa+LMn9SZ5Icl+SpX3brE+yJ8nuJGtH2QFpFH709N4Z\n62+47FdPbEOkERhkxH8I+K2qeivwL4DfTPIWYB2wtaouBrYB6wGSXAJcD6wCrgZuTzKv+0pIkoZn\n1uCvqgNVtaNZfgHYDawArgU2NattAq5rlq8B7qqqQ1W1F9gDrBlyuyVJx2lOc/xJzgdWA98Ezqmq\nKei9OQBnN6stB/onSPc3NUnSIrBk0BWTnAXcDXy0ql5IMv2eynO+x/LExMTR5U6nQ6fTmesuJGms\ndbtdut3uUPc50P34kywB/hz4y6r6dFPbDXSqairJucBfVdWqJOuAqqqNzXr3Ahuq6qFp+/R+/Fq0\nfrD763zv6390TP0dN35mAVoj/cSJvB//54HJI6Hf2AJ8oFl+P/C1vvoNSU5LcgFwIfDwfBopSRqe\nWad6klwBvA/YlWQ7vSmdjwMbgc1JPgg8Se9MHqpqMslmYBI4CNzk0F6SFo9Zg7+qHgROfZmX3/My\n29wK3DqPdkmSRsQrdyWpZQx+SWoZg1+SWsbgl6SWMfglqWUMfmkGP35u6pjakp89awFaIg2fwS/N\nYOqxB46pnfWGixagJdLwGfyS1DIGvyS1jMEvSS1j8EtSyxj8ktQyBr8ktYzBL0ktY/BLUssY/JLU\nMga/JLWMwS9JLWPwS1LLzBr8SVYk2Zbk8SS7kvy7pr4hyb4kjzaPq/q2WZ9kT5LdSdaOsgOSpLmZ\n9cvWgUPAb1XVjiRnAd9OcuTWhZ+qqk/1r5xkFXA9sApYAWxNclFV1TAbLo1KHX5poZsgjdSsI/6q\nOlBVO5rlF4DdwPLm5cywybXAXVV1qKr2AnuANcNprjR6T379zhnrK/75vz7BLZFGY05z/EnOB1YD\nDzWlm5PsSPLZJEub2nLgqb7N9vOTNwrppHX6q1+70E2QhmKQqR4Ammmeu4GPVtULSW4H/ktVVZL/\nCnwS+NBcDj4xMXF0udPp0Ol05rK5JI29brdLt9sd6j4zyNR7kiXAnwN/WVWfnuH1lcA9VXVpknVA\nVdXG5rV7gQ1V9dC0bZz216K096+/yDNPPHhM/R03fmYBWiP9tCRU1UzT7AMbdKrn88Bkf+gnObfv\n9X8F/G2zvAW4IclpSS4ALgQenk8jJUnDM+tUT5IrgPcBu5JsBwr4OPDeJKuBw8Be4MMAVTWZZDMw\nCRwEbnJoL0mLx6zBX1UPAqfO8NK9r7DNrcCt82iXJGlEvHJXklrG4JekljH4JallDH5JahmDX5Ja\nxuCXpJYx+CWpZQx+SWoZg1+a5oUDf3dM7TUXXLYALZFGw+CXpvnxc1PH1F71+gsWoCXSaBj80iDm\ndS9EaXEx+KVBeJtBjRGDXxqEI36NEYNfGoQjfo2Rgb96UTpZPffcc2zbtm3g9d80Q+3xx/+Wv/mf\nLw60/dKlS7nyyisHPp50og301YsjObBfvagTZOfOnaxevXrg9b/1mRuPqf3unz7EF+/fOdD2l156\nKTt3DrauNFfD+OpFR/zSDLZOvffo8juWbV3AlkjD5xy/NM3/ePrf8P8On3X08eAz1y10k6ShMvil\naf7vS68+pvb3P/qFBWiJNBqzBn+S05M8lGR7kl1JNjT1ZUnuT/JEkvuSLO3bZn2SPUl2J1k7yg5I\nJ8LPnvLCQjdBGppZg7+qfgz8UlVdBqwGrk6yBlgHbK2qi4FtwHqAJJcA1wOrgKuB25N4FrROGivP\nnPyp5+Ellp/x9wvUGmn4Bvpwt6qOnMd2erNNAdcC72rqm4AuvTeDa4C7quoQsDfJHmAN8NDwmi2N\nzu/87r/n+YOvZeU5S7ntxvdw5qnPc/tCN0oaooGCP8kpwLeBfwL8XlU9kuScqpoCqKoDSc5uVl8O\n/E3f5vub2jFuvPHY0+akYXvmmWfmtP6efc8CzzI1BVfe/K05H2/fvn3+bmtRG3TEfxi4LMnPAV9N\n8laOvZZxziflv/GNbzy63Ol06HQ6c92FNKudO3fyla985YQdb8WKFdxxxx0n7Hgab91ul263O9R9\nzuk8/qp6PkkXuAqYOjLqT3Iu8HSz2n7gvL7NVjS1Y0xMTMy5wZLUJtMHxZ/4xCfmvc9Bzup53ZEz\ndpKcAfwysBvYAnygWe39wNea5S3ADUlOS3IBcCHw8LxbKkkaikFG/G8ANjXz/KcAX66qv0jyTWBz\nkg8CT9I7k4eqmkyyGZgEDgI3eW8GSVo8Zg3+qtoFvH2G+rPAe15mm1uBW+fdOknS0HnlriS1jMEv\nSS1j8EtSy3hbZo29lStXcvfdd5+w4y1dunT2laQF5BexSNJJZBhfxOJUjyS1jMEvSS1j8EtSyxj8\nktQyBr8ktYzBL0ktY/BLUssY/JLUMga/JLWMwS9JLWPwS1LLGPyS1DIGvyS1zCBftn56koeSbE+y\nK8mGpr4hyb4kjzaPq/q2WZ9kT5LdSdaOsgOSpLkZ6LbMSc6sqheTnAo8CHwEuBr4P1X1qWnrrgK+\nBLwTWAFsBS6afg9mb8ssSXN3wm7LXFUvNoun0/vyliOJPdPBrwXuqqpDVbUX2AOsmU8jJUnDM1Dw\nJzklyXbgAPBAVT3SvHRzkh1JPpvkyNcOLQee6tt8f1OTJC0Cg474D1fVZfSmbtYkuQS4HXhzVa2m\n94bwydE1U5I0LHP6zt2qej5JF7hq2tz+7wP3NMv7gfP6XlvR1I4xMTFxdLnT6dDpdObSHEkae91u\nl263O9R9zvrhbpLXAQer6rkkZwD3AbcBj1bVgWadjwHvrKr3Nn8N3AlcTm+K5wH8cFeShmIYH+4O\nMuJ/A7ApySn0poa+XFV/keSLSVYDh4G9wIcBqmoyyWZgEjgI3GTCS9LiMdDpnCM5sCN+SZqzE3Y6\npyRpfBj8ktQyBr8ktYzBL0ktY/BLUssY/JLUMga/JLWMwS9JLWPwS1LLGPyS1DIGvyS1jMEvSS1j\n8EtSyxj8ktQyBr8ktYzBL0ktY/BLUssY/JLUMga/JLXMwMGf5JQkjybZ0jxfluT+JE8kuS/J0r51\n1yfZk2R3krWjaLgk6fjMZcT/UWCy7/k6YGtVXQxsA9YDJLkEuB5YBVwN3J5kxi8G7na7x9Hkk8M4\n9w3s38nO/rXbQMGfZAXwq8Bn+8rXApua5U3Adc3yNcBdVXWoqvYCe4A1M+13nH8449w3sH8nO/vX\nboOO+P878B+B6qudU1VTAFV1ADi7qS8Hnupbb39TkyQtArMGf5JfA6aqagcw45RNo17hNUnSIpGq\nV87rJL8D/DpwCDgDeDXwVeCfAZ2qmkpyLvBXVbUqyTqgqmpjs/29wIaqemjafn2jkKTjUFWvNAif\n1azB/1MrJ+8C/kNVXZPkvwHPVNXGJLcAy6pqXfPh7p3A5fSmeB4ALqq5HEiSNDJL5rHtbcDmJB8E\nnqR3Jg9VNZlkM70zgA4CNxn6krR4zGnEL0k6+S3IlbtJrkrynSTfbaaJTjpJPpdkKsljfbWxuKgt\nyYok25I8nmRXko809XHp3+lJHkqyvenfhqY+Fv07YpwvukyyN8nO5mf4cFMbp/4tTfInTXsfT3L5\nUPtXVSf0Qe/N5u+AlcDPADuAt5zodgyhH/8SWA081lfbCPynZvkW4LZm+RJgO72ptfOb/meh+/AK\nfTsXWN0snwU8AbxlXPrXtPnM5t9TgW/Su9ZkbPrXtPtjwB8BW8bp97Np8z/Q+1yxvzZO/fsD4Dea\n5SXA0mH2byFG/GuAPVX1ZFUdBO6idzHYSaWqvgH8cFp53he1LQZVdaB6p+9SVS8Au4EVjEn/AKrq\nxWbxdHr/YYox6t+oLrpcRMKxMxZj0b8kPwf8YlV9AaBp93MMsX8LEfzTL/Dax/hc4HV2jdlFbUnO\np/eXzTcZo4v2mmmQ7cAB4IGqeoQx6h/jf9FlAQ8keSTJh5rauPTvAuB/JflCM1V3R5IzGWL/vDvn\naJ3Un5wnOQu4G/hoM/Kf3p+Ttn9VdbiqLqP3l8yaJG9lTPrXkosur6iqt9P7q+Y3k/wiY/Lzo/cX\n6NuB32v6+CN690YbWv8WIvj3A2/qe76iqY2DqSTnADQXtT3d1PcD5/Wtt+j7nGQJvdD/w6r6WlMe\nm/4dUVXPA13gKsanf1cA1yT5B+CPgSuT/CFwYEz6R1V9v/n3B8Cf0ZvaGJef3z7gqar6VvP8T+m9\nEQytfwsR/I8AFyZZmeQ04AZgywK0YxjCT4+otgAfaJbfD3ytr35DktOSXABcCDx8ohp5nD4PTFbV\np/tqY9G/JK87ckZEkjOAX6b3OcZY9K+qPl5Vb6qqN9P7/7Wtqv4tcA9j0L8kZzZ/jZLkVcBaYBfj\n8/ObAp5K8k+b0ruBxxlm/xboE+ur6J0psgdYt9CfoB9nH74E/CPwY+B7wG8Ay4CtTd/uB17Tt/56\nep+27wbWLnT7Z+nbFcBL9M642g482vzMfn5M+ve2pk87gMeA/9zUx6J/0/r6Ln5yVs9Y9I/eHPiR\n381dRzJkXPrXtPcX6A2SdwBfoXdWz9D65wVcktQyfrgrSS1j8EtSyxj8ktQyBr8ktYzBL0ktY/BL\nUssY/JLUMga/JLXM/wcfPIOjC8dGDQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ab43518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(env.render('rgb_array'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic agent setup\n",
    "Here we define a simple agent that maps game images into Qvalues using shallow neural network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import lasagne\n",
    "from lasagne.layers import InputLayer,DenseLayer\n",
    "\n",
    "\n",
    "inp = InputLayer((None,len(env.reset())),)\n",
    "h1 = DenseLayer(inp,100,nonlinearity= lasagne.nonlinearities.tanh,name=\"h1\")\n",
    "h2 = DenseLayer(h1,100, nonlinearity= lasagne.nonlinearities.tanh,name=\"h2\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#a layer that predicts Qvalues\n",
    "qvalues_layer = DenseLayer(h2,env.action_space.n,nonlinearity=lambda x:x,name='q')\n",
    "\n",
    "#To pick actions, we use an epsilon-greedy resolver (epsilon is a property)\n",
    "from agentnet.resolver import EpsilonGreedyResolver\n",
    "action_layer = EpsilonGreedyResolver(qvalues_layer)\n",
    "\n",
    "action_layer.epsilon.set_value(np.float32(0.05))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Finally, agent\n",
    "We declare that this network is and MDP agent with such and such inputs, states and outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from agentnet.agent import Agent\n",
    "#all together\n",
    "agent = Agent(observation_layers=inp,\n",
    "              policy_estimators=qvalues_layer,\n",
    "              action_layers=action_layer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[h1.W, h1.b, h2.W, h2.b, q.W, q.b]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Since it's a single lasagne network, one can get it's weights, output, etc\n",
    "weights = lasagne.layers.get_all_params(action_layer,trainable=True)\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create and manage a pool of atari sessions to play with\n",
    "\n",
    "* To make training more stable, we shall have an entire batch of game sessions each happening independent of others\n",
    "* Why several parallel agents help training: http://arxiv.org/pdf/1602.01783v1.pdf\n",
    "* Alternative approach: store more sessions: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "action_layer.epsilon.set_value(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:28:28,381] Making new env: CartPole-v0\n",
      "INFO (theano.gof.compilelock): Waiting for existing lock by process '54912' (I am process '84804')\n",
      "[2017-01-06 12:28:39,347] Waiting for existing lock by process '54912' (I am process '84804')\n",
      "INFO (theano.gof.compilelock): To manually release the lock, delete /Users/alexajax/.theano/compiledir_Darwin-16.4.0-x86_64-i386-64bit-i386-3.5.2-64/lock_dir\n",
      "[2017-01-06 12:28:39,350] To manually release the lock, delete /Users/alexajax/.theano/compiledir_Darwin-16.4.0-x86_64-i386-64bit-i386-3.5.2-64/lock_dir\n"
     ]
    }
   ],
   "source": [
    "from agentnet.experiments.openai_gym.pool import EnvPool\n",
    "\n",
    "pool = EnvPool(agent,GAME, N_AGENTS,max_size=10000)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['left' 'left' 'stop' 'left' 'stop' 'left' 'left']]\n",
      "[[ 1.  1.  1.  1.  1.  1.  0.]]\n",
      "CPU times: user 8.33 ms, sys: 3.88 ms, total: 12.2 ms\n",
      "Wall time: 9.32 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#interact for 7 ticks\n",
    "_,action_log,reward_log,_,_,_  = pool.interact(7)\n",
    "\n",
    "\n",
    "print(action_names[action_log])\n",
    "print(reward_log)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/alexajax/anaconda3/lib/python3.5/site-packages/agentnet/environment/session_pool.py:292: UserWarning: Warning! Appending sessions to empty or broken pool. Old pool sessions, if any, are disposed.\n",
      "  warn(\"Warning! Appending sessions to empty or broken pool. Old pool sessions, if any, are disposed.\")\n"
     ]
    }
   ],
   "source": [
    "#load first sessions (this function calls interact and remembers sessions)\n",
    "pool.update(SEQ_LENGTH,append=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#pool.experience_replay.observations[0].get_value()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 1, 1, 1, 0, 0, 0, 1, 1]], dtype=int32)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pool.experience_replay.actions[0].get_value()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-learning\n",
    "* An agent has a method that produces symbolic environment interaction sessions\n",
    "* Such sessions are in sequences of observations, agent memory, actions, q-values,etc\n",
    "  * one has to pre-define maximum session length.\n",
    "\n",
    "* SessionPool also stores rewards (Q-learning objective)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/alexajax/anaconda3/lib/python3.5/site-packages/agentnet/agent/mdp_agent.py:142: UserWarning: optimize_experience_replay is deprecated and will be removed in 1.0.2. Use experience_replay parameter.\n",
      "  warn(\"optimize_experience_replay is deprecated and will be removed in 1.0.2. Use experience_replay parameter.\")\n"
     ]
    }
   ],
   "source": [
    "#get agent's Qvalues obtained via experience replay\n",
    "replay = pool.experience_replay.sample_session_batch(100,replace=True)\n",
    "\n",
    "_,_,_,_,qvalues_seq = agent.get_sessions(\n",
    "    replay,\n",
    "    session_length=SEQ_LENGTH,\n",
    "    optimize_experience_replay=True,\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#get reference Qvalues according to Qlearning algorithm\n",
    "from agentnet.learning import qlearning\n",
    "\n",
    "#crop rewards to [-1,+1] to avoid explosion.\n",
    "#import theano.tensor as T\n",
    "#rewards = T.maximum(-1,T.minimum(rewards,1))\n",
    "\n",
    "#loss for Qlearning = (Q(s,a) - (r+gamma*Q(s',a_max)))^2\n",
    "\n",
    "elwise_mse_loss = qlearning.get_elementwise_objective(qvalues_seq,\n",
    "                                                      replay.actions[0],\n",
    "                                                      replay.rewards,\n",
    "                                                      replay.is_alive,\n",
    "                                                      gamma_or_gammas=0.99,)\n",
    "\n",
    "#compute mean over \"alive\" fragments\n",
    "loss = elwise_mse_loss.sum() / replay.is_alive.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Compute weight updates\n",
    "updates = lasagne.updates.sgd(loss,weights,learning_rate=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#compile train function\n",
    "import theano\n",
    "train_step = theano.function([],loss,updates=updates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "theano.printing.pydotprint(train_step,\"qv.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:30:49,505] Making new env: CartPole-v0\n",
      "[2017-01-06 12:30:49,522] Attempted to wrap env <CartPoleEnv instance> after .configure() was called. All wrappers must be applied before calling .configure()\n",
      "[2017-01-06 12:30:49,525] Clearing 4 monitor files from previous run (because force=True was provided)\n",
      "[2017-01-06 12:30:49,530] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.0.84804.video000000.mp4\n",
      "[2017-01-06 12:30:50,806] Finished writing results. You can upload them to the scoreboard via gym.upload('/Users/alexajax/Downloads/goto/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 26 timesteps with reward=26.0\n"
     ]
    }
   ],
   "source": [
    "#for MountainCar-v0 evaluation session is cropped to 200 ticks\n",
    "untrained_reward = pool.evaluate(save_path=\"./records\",record_video=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"640\" height=\"480\" controls>\n",
       "  <source src=\"records/openaigym.video.0.54912.video000000.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "video_path=\"records/openaigym.video.0.54912.video000000.mp4\"\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(video_path))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#starting epoch\n",
    "epoch_counter = 1\n",
    "\n",
    "#full game rewards\n",
    "rewards = {epoch_counter:untrained_reward}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [00:05<00:00, 175.69it/s]\n"
     ]
    }
   ],
   "source": [
    "#pre-fill pool\n",
    "from tqdm import tqdm\n",
    "for i in tqdm(range(1000)):\n",
    "    pool.update(SEQ_LENGTH,append=True,)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1001, 10, 4)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pool.experience_replay.observations[0].eval().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  5%|▍         | 498/10000 [00:36<10:07, 15.64it/s][2017-01-06 12:31:33,087] Making new env: CartPole-v0\n",
      "[2017-01-06 12:31:33,100] Attempted to wrap env <CartPoleEnv instance> after .configure() was called. All wrappers must be applied before calling .configure()\n",
      "[2017-01-06 12:31:33,102] Clearing 8 monitor files from previous run (because force=True was provided)\n",
      "[2017-01-06 12:31:33,107] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.1.84804.video000000.mp4\n",
      "[2017-01-06 12:31:34,596] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.1.84804.video000001.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 67 timesteps with reward=67.0\n",
      "Episode finished after 71 timesteps with reward=71.0\n",
      "Episode finished after 81 timesteps with reward=81.0\n",
      "Episode finished after 65 timesteps with reward=65.0\n",
      "Episode finished after 76 timesteps with reward=76.0\n",
      "Episode finished after 72 timesteps with reward=72.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:31:36,213] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.1.84804.video000008.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 79 timesteps with reward=79.0\n",
      "Episode finished after 77 timesteps with reward=77.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:31:37,729] Finished writing results. You can upload them to the scoreboard via gym.upload('/Users/alexajax/Downloads/goto/records')\n",
      "  5%|▌         | 500/10000 [00:41<2:00:25,  1.31it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 77 timesteps with reward=77.0\n",
      "Episode finished after 65 timesteps with reward=65.0\n",
      "Current score(mean over 10) = 73.000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 10%|▉         | 999/10000 [01:20<12:11, 12.30it/s][2017-01-06 12:32:17,382] Making new env: CartPole-v0\n",
      "[2017-01-06 12:32:17,398] Attempted to wrap env <CartPoleEnv instance> after .configure() was called. All wrappers must be applied before calling .configure()\n",
      "[2017-01-06 12:32:17,400] Clearing 8 monitor files from previous run (because force=True was provided)\n",
      "[2017-01-06 12:32:17,404] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.2.84804.video000000.mp4\n",
      "[2017-01-06 12:32:19,858] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.2.84804.video000001.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 109 timesteps with reward=109.0\n",
      "Episode finished after 106 timesteps with reward=106.0\n",
      "Episode finished after 107 timesteps with reward=107.0\n",
      "Episode finished after 112 timesteps with reward=112.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:32:22,304] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.2.84804.video000008.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 111 timesteps with reward=111.0\n",
      "Episode finished after 111 timesteps with reward=111.0\n",
      "Episode finished after 105 timesteps with reward=105.0\n",
      "Episode finished after 108 timesteps with reward=108.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:32:24,642] Finished writing results. You can upload them to the scoreboard via gym.upload('/Users/alexajax/Downloads/goto/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 107 timesteps with reward=107.0\n",
      "Episode finished after 109 timesteps with reward=109.0\n",
      "Current score(mean over 10) = 108.500\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 15%|█▍        | 1499/10000 [02:04<13:08, 10.79it/s][2017-01-06 12:33:01,379] Making new env: CartPole-v0\n",
      "[2017-01-06 12:33:01,395] Attempted to wrap env <CartPoleEnv instance> after .configure() was called. All wrappers must be applied before calling .configure()\n",
      "[2017-01-06 12:33:01,397] Clearing 13 monitor files from previous run (because force=True was provided)\n",
      "[2017-01-06 12:33:01,401] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.3.84804.video000000.mp4\n",
      "[2017-01-06 12:33:05,688] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.3.84804.video000001.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:33:11,044] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.3.84804.video000008.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 15%|█▍        | 1499/10000 [02:20<13:14, 10.70it/s][2017-01-06 12:33:17,490] Finished writing results. You can upload them to the scoreboard via gym.upload('/Users/alexajax/Downloads/goto/records')\n",
      " 15%|█▌        | 1500/10000 [02:20<11:37:56,  4.93s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Current score(mean over 10) = 200.000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 20%|█▉        | 1998/10000 [02:55<08:05, 16.47it/s][2017-01-06 12:33:52,470] Making new env: CartPole-v0\n",
      "[2017-01-06 12:33:52,484] Attempted to wrap env <CartPoleEnv instance> after .configure() was called. All wrappers must be applied before calling .configure()\n",
      "[2017-01-06 12:33:52,486] Clearing 6 monitor files from previous run (because force=True was provided)\n",
      "[2017-01-06 12:33:52,489] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.4.84804.video000000.mp4\n",
      "[2017-01-06 12:33:56,322] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.4.84804.video000001.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:34:00,310] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.4.84804.video000008.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n",
      "Episode finished after 200 timesteps with reward=200.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:34:05,551] Finished writing results. You can upload them to the scoreboard via gym.upload('/Users/alexajax/Downloads/goto/records')\n",
      " 20%|██        | 2002/10000 [03:09<3:11:56,  1.44s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current score(mean over 10) = 200.000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 25%|██▍       | 2499/10000 [03:55<14:02,  8.90it/s][2017-01-06 12:34:52,208] Making new env: CartPole-v0\n",
      "[2017-01-06 12:34:52,227] Attempted to wrap env <CartPoleEnv instance> after .configure() was called. All wrappers must be applied before calling .configure()\n",
      "[2017-01-06 12:34:52,230] Clearing 11 monitor files from previous run (because force=True was provided)\n",
      "[2017-01-06 12:34:52,235] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.5.84804.video000000.mp4\n",
      "[2017-01-06 12:34:57,108] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.5.84804.video000001.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 99 timesteps with reward=99.0\n",
      "Episode finished after 102 timesteps with reward=102.0\n",
      "Episode finished after 101 timesteps with reward=101.0\n",
      "Episode finished after 100 timesteps with reward=100.0\n",
      "Episode finished after 99 timesteps with reward=99.0\n",
      "Episode finished after 101 timesteps with reward=101.0\n",
      "Episode finished after 99 timesteps with reward=99.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:35:00,135] Starting new video recorder writing to /Users/alexajax/Downloads/goto/records/openaigym.video.5.84804.video000008.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 99 timesteps with reward=99.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-01-06 12:35:02,670] Finished writing results. You can upload them to the scoreboard via gym.upload('/Users/alexajax/Downloads/goto/records')\n",
      " 25%|██▌       | 2500/10000 [04:06<6:46:37,  3.25s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 98 timesteps with reward=98.0\n",
      "Episode finished after 100 timesteps with reward=100.0\n",
      "Current score(mean over 10) = 99.800\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 27%|██▋       | 2662/10000 [04:23<14:53,  8.22it/s]"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-33-213ebaf2c12f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      8\u001b[0m     \u001b[0mpool\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mSEQ_LENGTH\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m         \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/alexajax/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    864\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    865\u001b[0m             \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 866\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0moutput_subset\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    867\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput_subset\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_subset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    868\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/alexajax/anaconda3/lib/python3.5/site-packages/theano/gof/op.py\u001b[0m in \u001b[0;36mrval\u001b[0;34m(p, i, o, n)\u001b[0m\n\u001b[1;32m    905\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mparams\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mgraph\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNoParams\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    906\u001b[0m             \u001b[0;31m# default arguments are stored in the closure of `rval`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 907\u001b[0;31m             \u001b[0;32mdef\u001b[0m \u001b[0mrval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode_input_storage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode_output_storage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    908\u001b[0m                 \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    909\u001b[0m                 \u001b[0;32mfor\u001b[0m \u001b[0mo\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "\n",
    "#the loop may take eons to finish.\n",
    "#consider interrupting early.\n",
    "for i in tqdm(range(10000)):    \n",
    "    \n",
    "    \n",
    "    #train\n",
    "    pool.update(SEQ_LENGTH,append=True,)\n",
    "    for i in range(5):\n",
    "        loss = train_step()\n",
    "        \n",
    "    \n",
    "        \n",
    "\n",
    "    ##record current learning progress and show learning curves\n",
    "    if epoch_counter%500 ==0:\n",
    "        n_games = 10\n",
    "        action_layer.epsilon.set_value(0)\n",
    "        rewards[epoch_counter] = pool.evaluate( record_video=True,n_games=n_games,verbose=True)\n",
    "        print(\"Current score(mean over %i) = %.3f\"%(n_games,np.mean(rewards[epoch_counter])))\n",
    "        action_layer.epsilon.set_value(0.05)\n",
    "    \n",
    "    \n",
    "    epoch_counter  +=1\n",
    "\n",
    "    \n",
    "# Time to drink some coffee!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "iters,session_rewards=zip(*sorted(rewards.items(),key=lambda (k,v):k))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f9414452c10>]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEACAYAAACgS0HpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmcHNV1779HIwbtK9LMCG1IRkGIzWITJuBhkQyB2CKP\n+JGPDY5N3vOCHTveAG9I8RJjOw+ME2zHIcFLwI4JBowBATaDjZFAC1rBktiEZiShfQWkkea+P04X\n04xmerq7lltVfb6fz3ym53Z11ema7vrVWe654pzDMAzDMHqjj28DDMMwjGxggmEYhmGUhQmGYRiG\nURYmGIZhGEZZmGAYhmEYZWGCYRiGYZRFKMEQkctFZKWIHBKR6V2eu15E1orIcyIyq2h8uogsF5E1\nInJzmOMbhmEYyRHWw1gBXAY8XjwoIlOB9wJTgYuBW0VECk9/H7jaOTcFmCIi7wppg2EYhpEAoQTD\nObfaObcWkC5PvQf4uXPuoHPuZWAtcIaINAKDnXMLC9v9BJgdxgbDMAwjGeLKYRwNrC/6u60wdjTQ\nWjTeWhgzDMMwUk7f3jYQkUeAhuIhwAFfdM79Oi7DDMMwjHTRq2A452ZWsd82YFzR32MLYz2Nd4uI\nWKMrwzCMKnDOdU0VhCbKkFSxcfcBV4hIvYgcA7wNeNo5twnYJSJnFJLgVwH3ltqpc85+nOOGG27w\nbkNafuxc2Lmwc1H6Jy7CltXOFpH1wAzgfhF5EMA59yzw38CzwAPAx1znu7gGuA1YA6x1zj0UxgbD\nMAwjGXoNSZXCOXcPcE8Pz/0T8E/djC8GTgxzXMMwDCN5bKZ3RmhubvZtQmqwc9GJnYtO7FzEj8QZ\n7wqLiLg022cYhpFGRASX8qS3YRiGkWNMMAzDMIyyMMEwDMMwysIEwzAMwygLEwzDMAyjLEwwDMMw\njLIwwTAMwzDKwgTDMAzDKAsTDMMwDKMsTDBi5rXX4MAB31YYhmGExwQjZj7yEfjhD31bYRiGEZ5Q\n3WqN0nR0wIMPwuDBvi0xDMMIj3kYMbJkCWzdCi+84NsSwzCM8JhgxMhDD8FFF8GLL/q2xDAMIzwm\nGDEyb57mMF55BQ4d8m2NYRhGOEwwYmLXLli6FGbNglGjYP163xYZhmGEwwQjJn77Wzj7bOjfHyZP\ntjyGYRjZxwQjJubNg3e9Sx+bYBiGkQdMMGLAOU14B4IxaZIJhmEY2ccEIwZWr1bRmDpV/5482Sql\nDMPIPiYYMRB4F1JYgt1CUoZh5AETjBiYN0/nXwQEguGcP5sMwzDCIi7FVzERcWm2rztefx0aGnTu\nxbBhnePDh8PatXDUUf5sMwyjNhARnHMS9X7Nw4iYP/wBTjzxrWIBFpYy0sfLL8O11/q2wsgSJhgR\n0zUcFWCVUkbauPFG+NnPfFthZAkTjIgpnn9RjHkYRpp49VX4xS9gyxZrW2OUjwlGhKxfD5s2wamn\nHv6cldYaaeJ734MrroChQ7WjsmGUgwlGhDz8MMycCXV1hz9nHoaRFvbu1UW9PvMZaGqCjRt9W2Rk\nBROMCCme3d0VEwwjLfzoR3D++fqZNMEwKsEEIyIOHtSGgz0JxtFHw/btWnZrGL5ob4ebboLPf17/\nNsEwKsEEIyIWLoRx4/QL2B11dTBhguUxDL/ceSdMmdKZZzPBiI758+H22/M9QdcEIyJKhaMCrLTW\n8Ilz8K1vdXoXYIIRJb/8JXzsYzB7tlaf5RETjIjoaf5FMZbHMHzywANwxBFamBFgghEdra3wgx9o\n09GTT4YHH/RtUfSYYETAtm3w7LO6YFIprLTW8EngXUhRwwgTjOhobdUowje/CXfcAR/+MHziE/nK\nW4YSDBG5XERWisghEZleNH6hiCwSkWUislBEzit6brqILBeRNSJyc5jjp4VHH4Vzz4Ujjyy9nXkY\nhi8WLND+Zn/9128dN8GIjrY2GDtWHzc3w7JlOsfltNN0ueY8ENbDWAFcBjzeZXwLcKlz7mTgb4Gf\nFj33feBq59wUYIqI9BL5Tz/lhKPABMPwx7e+pfMu+vZ963hTk042zXOiNgkOHVLhHTOmc2z4cPU0\nrr9ew4Df+Q50dPizMQpCCYZzbrVzbi0gXcaXOec2FR6vAvqJyBEi0ggMds4tLGz6E2B2GBt841zP\n7UC6cswxsG6dtWIwkmX1avjjH+FDHzr8uQEDoL4edu5M3q6Au+7SLglZZvNmGDFCz2UxIvD+92sV\n5b33qnC0tvqxMQpiz2GIyOXAEudcO3A0UHy6WgtjmWXlSujXD972tt637d9f25tn+QNjZI9vfxuu\nuUbFoTt8h6Vuugl+/3t/x4+C1tbOcFR3TJwILS1wwQVa0vzLXyZlWbT07W0DEXkEaCgeAhzwRefc\nr3t57TTgn4CZpbYrxZw5c9583NzcTHNzc7W7ioXAu5AyO88HpbUTJsRrl2EAbNgAd9+ta7H0RCAY\nxx+fnF3FtLVpWCzL9CYYoHOxvvAF9TLe9z74zW/glltgyJDwx29paaGlpSX8jnqhV8FwzlV1sReR\nscDdwJXOuZcLw23AuKLNxhbGeqRYMNLIQw9pJUS5BJVS558fn02GEfDd78KVV8LIkT1v49PD6OhQ\nUasFwQg4/XR45hn49KfhlFO0xfw73hHu+F1vpufOnRtuhz0QZUjqzXtsERkK3A9c65xbEIwX8hq7\nROQMERHgKuDeCG1IlH374KmnKrv4W+LbSIpdu+Df/x3+4R9Kb+dTMLZu1XYlr77q5/hRUYlgAAwc\nqA0gb7oJ/uqv4Ctf0fOQdsKW1c4WkfXADOB+EQmmqnwcmAx8RUSeEZElIhIsTnoNcBuwBljrnHso\njA0+aWnReOTgweW/xpdgPPecJdtrjR/+EC6+WOPnpfApGG2F+EIteRjFvOc9WnK7cCGccw48/3z0\ntkVJryGpUjjn7gHu6Wb868DXe3jNYuDEMMdNC+VWRxXjQzDWrNFlY0eMgEsugXe/G2bN0rscI5/s\n36/hqAce6H3bpiZYvDh+m7qjrU0bc9aqYAA0Nur/6V/+Bc46S5d5Pu64aO2LCpvpHYJy518UEwhG\nknXvzzyjIrFwoXpEt96qF4lLL9VW1zZxK3/87Gdw0knaoqI3fHsY06fXtmCAFs184hN6A/r009HZ\nFTUmGFXy0ktau17OF7KYESNULLZvj8eu7li2TO2cMAE+/nF45BGte3//++Gxx2DaNJgxA77xDVi1\nyiZxZZ2ODi2lLW4yWArfgnHyybBjRzZi+N3hXKenFJZJk9LdPsgEo0qCcFSfCs+gSPJhqUAwihk6\nVJfovOMOvbv7+tf19yWX6JyST39aczQHDyZnpxEN992npZrlVqD7Fozx43V+UlY7vG7dCoMG6Tyr\nsJhg5JRq8hcBSTch7E4wiqmv1wlFt9yintPdd2tbg89+FhoatCzzrrtg9+7kbDaqwzm48cbDmwyW\nYuhQvbvfty9e27ojuDNvbMxupVTYcFQxxxyj38G0YoJRBe3tGsqZWeV0xCQ9jG3bdA3n3iplAkRU\nXL78ZVi0SMXmHe+A227TL/aMGdobZ948PxcYozRPPKF3vJddVv5rRPx5GcWCkdU8RpSCYR5GDpk/\nX8M2o0dX9/okBWPZMk1+lnu32ZWxY+GjH9Xe/lu2aOvm+noNYTU0aEv3L31Jl6fNUxvnrHLjjeoZ\n1tVV9jpfgtHaqoLR0GCCAdq8cNu29H6XTDCqoJzV9UqRtGBUmpjviX79NC4+d672/tm8WR+DTjwa\nNQre+U644QbNf7zxRjTHNcpj5Uotj/3AByp/rQ/BeO01vTCOHGkeRkCwlPPLL0ezv6gxwaiCaspp\ni8mqYHRlwAC48EL42te0G+qmTdorZ/9+uO46FZDzz4evflVDJQcOxGOHoXz721qa2a9f5a/1IRht\nbXpHLWKCUUyaw1ImGBWyebNe7GfMqH4fY8dqnDkJtzNOwejKoEHqeX3zm7pgT1ubhkd274ZPfUrv\nJB/K7Lz+dLN+Pfz61xo+rAZfghFcaC3p3YkJRo54+GE47zxdG7la6uq0lDDuaoj2dl0L4YQT4j1O\nTwwZAn/xF3rnu2iRLuJzxx1+bMk7N90EH/ygVrdVgy/BCOYumIfRiQlGjggbjgpIorT2T3/SeGgU\n9eFRMGuWThq0iYHRsmMH3H57700GS+FbMLKa9HauM3EfFZMmpbe01gSjAjo61MMIk/AOSCKPkWQ4\nqhwmT1bxWrnStyX54tZbtfVLmLtc34KRVQ9j506NNlTSgLQ3jjnGPIxcsHQpDBtW/pyGUtSiYECn\nl2FEw+uvw/e+B5/7XLj9+BaMYcP0vaS1nLQnog5HQadgpNETN8GogKjCUVC7gjFzpglGlPz4x7og\nz7Rp4fZz1FFanJBkJVuxYASVUllLfMchGEOHaqVbGlulmGBUQNj5F8UES7XGhXPqEaVNMM4/X0ts\nbY5GeA4dgu98B669Nvy++vTRiahJhoW6NuwzwegkrYlvE4wy2b0blizRiWlRMGmSTs6Ja1GjTZs0\n5zJmTDz7r5bhw/Vu+MknfVuSfe6+Wy/yZ58dzf6SDEsdOqSf0eLPZxbzGCYYRrc89pjOvYhq0aEB\nA7TVeVvJFc2rJwhHVdsSJE4sLBUNgXcR1f84ScHYvFnzFvX1nWNZrJQywcgJe/dqD6V/+7do7uKj\nDEcFxFlam8b8RcDMmVptZlTP1q1aNn3ppdHtM0nB6G79CPMwOjHBSJh16/RL9bOfaVLwj3+sfl/O\nqWBElfAOiDPxnWbBmDFD1y7eutW3JdllwQI488zKmwyWwgSjcuISjLS2Oc+tYLS1wfHHw+OPa8nh\nFVfoug4bNlS+r7VrddZ02EqUrtSqYNTXw7nnaodbozrmz9e281FiglE55mHkhGD2pQj8zd/Ac8/B\nuHEaprrxRm2QVy7z5un8gajzAXEJxhtv6Idt6tTo9x0VlscIx5NPwllnRbvPpAWj64U2a1VSu3dr\nuHvo0Oj3PW6cimfaGnbmVjC6fiAHDdI1qxcs0PDUCSfAb35T3r6inH9RTFyltatWwbHHwpFHRr/v\nqJg1S/MYaZyclHYOHtTeXGeeGe1+fXsYWUt6B9eYOApL+vbVfa9bF/2+w5Brweiuv8vb3qZrHt9y\ni65bfemlGnLqif37de2HCy+M3sa4PIw0h6MC/uzPVCzWrPFtSfZYvly7DQwbFu1+0yIYWbmJiLqH\nVFfSGJbKrWD09s+8+GJYsULnVZx1lq7fsGfP4ds98YTmLkaMiN7Go45Sl3bHjmj3u2wZnHJKtPuM\nGhELS1VLHOEo0Av2li3xzQ0qpjvBGDRIk/jdfQ/TSFz5iwATjATpLkbalfp6TYivWKF3Vscdp1VV\nxXc48+ZFX04bIBKPl5EFDwM6w1JGZTz5ZPQJb9AmesOHJ9OSoqcIQJYS3yYYOaISd7GpSXvy3HUX\n3Hwz/Pmf66xuiGf+RTFRC4Zz2RGMCy7QKrb2dt+WZIv58+PxMCCZsNTevfo/7y6klqXEd9yCkcbS\n2lwKxv79sGuXtk2ohLPOgqefhg99SBf+ueoq/VCcfno8dkL0grF+vbYQHzUqun3GxahR+v6fesq3\nJdlhwwYN2UyZEs/+kxCMwLvoLllsHkYn5mEkxIYN+sHvU8W769MHrr5aZ9GOGKGP+/aN3saAqCul\nsuJdBNis78oIvIu4Wr4kKRjdkaVKqSQE44UX0lUEkEvBiKJ6YdgwDU99+9vR2NQTUXsYWRMMWx+j\nMuIMR0EyglHq+2keRifDh+uNQdRFMWHIpWCUk/BOC7UuGGefrSvw7dzp25JsEFfCO8C3h5EVwXjt\nNdi3Tysd40IkfWGp3ApGnPXRUTJunHbujGp9iDSugVGKfv30Avi73/m2JP3s3683BHHm1EwwyqNU\nHiZKTDASIO4JNVHSty+MH69rY4Rl717N38SVEI0LC0uVx5IlWvodVYv97kiDYGShSirucFSACUYC\nZCkkBdGFpVas0P5RcSbp48Am8JVH3OEo8C8YWUl6JyUYaSutzaVgZMnDgOgEI2v5i4ATT1TvKE13\nUmkkKcGIuz1HqRu60aM1RNvREd/xo8A8jByRpRwGRFdam1XBsDYhveNcfC1BiunfX/NKcVXmHDyo\nM8kbG7t//sgjYfBg2L49nuNHhQlGFYjI5SKyUkQOicj0bp4fLyJ7ROTTRWPTRWS5iKwRkZvDHL87\nOjrUpU7bWtalqHUPA0wwemPdOhXWCRPiP1acYalXX4WRI7UNSU9kIfGdlGBMmKDHOngw/mOVQ1gP\nYwVwGfB4D8//M/BAl7HvA1c756YAU0Qk0sYbmzdrf/p+/aLca7xEIRgdHZrDOOmkaGxKmgsv1Eqp\nJBrfZZEgHJXEGu1xCkY53r8JRif19Xo+1q+P/1jlEEownHOrnXNrgcM+xiLyHuBFYFXRWCMw2Dm3\nsDD0E2B2GBu6krWEN6jb+fLL4eK2L76oM9OHD4/MrEQZM0YvJIsW+bYkncQ9Ya+YNAhG2iulkhIM\nSFdYKpYchogMBD4PzOWtYnI00Fr0d2thLDKylr8ALZMcNqy65WMDshyOCrCwVM8kkfAO8C0Yaa+U\n2r9fczyV9qqrlkwJhog8Usg5BD8rCr//ssTL5gA3Oedei8zSMslahVRA2LCUCUZ+2bcPVq+G6Ydl\nCeMhTsEo5/uZ9pBU0Kuuri6Z46VJMHqt2HfOzaxiv2cC/0tEvgUMBw6JyBvA3cC4ou3GAm2ldjRn\nzpw3Hzc3N9Pc3FzywFkMSUGnYLzzndW9ftkyuPLKaG1KmnPPhfe+V7uxDh7s25r0sHCh5qaSWnK3\nqUmPGQdtbdrWvhSNjZqPSytJhqNA52Lcd1/pbVpaWmhpaYndliineL0ZenLOnfvmoMgNwB7n3K2F\nv3eJyBnAQuAq4JZSOy0WjHJoba3+ouuTsKW1y5bBd74TnT0+GDhQ2148/rgunWsoSYajwH9IKu0e\nRtKCUY6H0fVmeu7cubHYErasdraIrAdmAPeLyINlvOwa4DZgDbDWOfdQGBu6ksUcBoQLSe3cCdu2\n6T6yjoWlDscEI12kUTCSIpSH4Zy7B7inl23mdvl7MXBimOOWIsshqWo/FMuX62zpatb/SBszZ+rC\nVYbinFZI/ehHyR0zLsFwLh9VUq2tMHFicscbNUqbk+7apVMGfJKDS8xbqcWkdx4S3gFvf7vOpWlt\n7X3bWmDNGhgyRC/iSTFkiM6H2bs32v3u3t25/1KMHKlVSGldujdpDyNoc56GnlK5Eozdu/UuxrcK\nV8OoUXDgQHXrQuRJMOrqNClqYSkl6XAU6AUqDi8j8P57m3xYV6frTGzZEu3xoyJpwYD0hKVyJRhJ\n9aiPA5HqvYysrYHRG5bH6CTJCXvFxCUY5Xr/ac5jmGDkhKyGowKqqZQ6eBCefVZzGHlh5kx49NH0\ndyxNAh8eBphg9ER7e+nmiXGRljbnuRKMrCa8A6rxMNas0S/hoEHx2OSDCRO0xcmyZb4t8cvOndp0\n0Ed/sDQIRhoT35s2afi4VPPEODAPIwayWlIbUI1g5Cl/UYyFpeCpp+C00/wsiOVbMNLaHsRHOApM\nMGIh6yGpakpr8yoYs2bBww/7tsIvvsJR4F8w0hqS8iUYEyeqt+m7m3OuBKMWQ1J5FYzmZr3Dfv11\n35b4I4kFk3oiDsGo5IbOBOOt9O+v5cZhGpRGQa4EI+sexvjx+iXZv7/81+RVMIYM0ff1hz/4tsQP\nhw7B00/DjBl+jm8eRvf4EgxIR1gqV4KRdQ+jb18YN07XxiiHLVv0Dnz8+FjN8kYth6VWrdKL9lFH\n+Tl+1ILR3q7taxoaytveBONwTDAi5MCBZHvUx0UlpbXLlmkFTRbnnZRDLSe+fYajQMMfe/ZU5u2W\nYuNG/W6Wm8BPa5WUT8E45hgTjMjYsEHvXpLqUR8XleQx8hqOCjj9dHjllXReOOJm/nx/CW/QvmRR\nVipVWsE4dKj2T0pbDsu3h+F7LkZuBCPr4agAE4xO+vaF887TSXy1hs8KqYAow1KVCoaIClaabhYO\nHdLzMWaMn+NbSCpCsj4HI6CS0tq8CwZoWKrW8hibN2t+aupUv3b4FAxIXx5j82adUJrUQlZdMcGI\nkKxXSAWU62EcOKCzvKdNi98mnwR5DOd8W5IcCxZodZTvdvVRC0alEYC0CYbPcBTo+di1S5fs9UVu\nBCMvIakgTtlbH6XnntMkWP/+ydjli8mT9Y7u2Wd9W5IcaQhHgXkYXfEtGH36+O8plRvByIuHMWiQ\nrmfd2xe1FsJRoLHsWiuv9V0hFeBbMNKWw/AtGOA/LJUbwciLhwHlhaWWLoVTTknGHt/UUnltezss\nWQJnnunbEv+CYR7G4ZhgRERekt5QnmDUiocBcP758MQT0c0JSDNLl+r/v7dV6ZIgKsFwrroIgAnG\n4VhIKgI6OnQehq9yt6jprVLKudoSjBEjtGLoySd9WxI/vhZM6o6oBGPHDqivr7wFvwnG4ZiHEQFb\nt2rcPy8J4N48jA0bNAGW9CIuPqmVsFRaEt6gOYStW8N3SK3W+zfBOBwTjAjIUzgKeheMwLvIa0uQ\n7jDBSJ6+fdW727w53H6q/X4GSe80lFQ7l47rTBCS8nVOciEYeamQCihXMGqJs87SeSfbtvm2JD5a\nW7UdxuTJvi3pJIqwVLUX2oEDtdXPnj3hjh8FW7eqPQMG+LUjqKL05XnlQjDyVCEF2qTtjTd0kk53\n1KJg1NfDOefAb3/r25L4CPIXafIcfQoGpCcslYZwVIDPsFQuBCNvHoZI6a61tSgYkP+wVJrCUQEm\nGIoJhpILwcibhwE9h6Vef12XajzuuORt8s2sWfluE5KmCqkAEwwlTYLhs825h+XloycNyaio6am0\nduVKmDJFQzS1xnHHafO3UaP0HBx7rP4OHh97rMaZs8jrr8OKFXDaab4teStNTeHbsoS5oTPBOJxJ\nk+D3v/dz7FwIRt5CUqCC8cwzh4/XajgKNFS3ZIlWzqxdq0nwtWvhzjv19/PPa1VPsZgEvydN8tdl\ntBwWL9ZGkr6Tql1pagqfNwpzQ5eW9iCtrbrOfBqYNAluv93PsXMhGHkNSd111+HjtSwYoKLR2Kg/\n55zz1uc6OvSLHQjJmjXw2GP6eN06vWgdeyxccgn8/d/7sb8n0hiOgvAhqf37YefO6lfCbGzUc+Ob\ntHkYFpKqkj17tP/OsGG+LYmWnpLey5bB7NnJ25MF+vTR9c3Hj4cLL3zrc+3tulb6qlXw/vfDxz/u\nv314MU8+CVdc4duKwwkrGBs36kW/2nNtIanDOfpoLfN94w3o1y/ZY6foK1MdgbubplLEKJgwQb9s\nBw50jjkHy5fXtodRLUccod7F7NmaB1m3zrdFnTiXzgopUMHYtKn6QoOw4eI0CEbQCystglFXpzdF\nL7+c/LFzIRhp+UdGyRFH6Bet+EOxbp0mdY86yptZueD449O1vsZLL+n/e9w435YcTr9+mlfZvr26\n14ctSEmDYOzcqbPeBw/2a0cxvsJSmReMPCa8A7qW1tZ6/iIqpk3T0FRaSKt3ERAmLBVWMEaP1uVq\ne1tQLE7S5F0E+Cqtzbxg5NXDgMNLa5cuNcGIgrR5GCYYPVNfr3f21Xo4UZBGwQhW5kyaXAiGeRhG\nJaTNw0hrhVSAT8EA/2GptAqGeRhVYCEpo1KmTtU10X2GOQL27NGy37e/3bclPWOCYYIREEowRORy\nEVkpIodEZHqX504SkScLzy8TkfrC+HQRWS4ia0Tk5jDHh3yHpIpLa3fv1i/NlCl+bcoDw4bpz/r1\nvi2Bp5/WpXbTPHPfBCN915hAMJJukxPWw1gBXAY8XjwoInXAT4H/65w7AWgG2gtPfx+42jk3BZgi\nIu8KY0DeQ1LBh2LFCg2l1NX5tiofHH98OsJS8+enO38B1QuGc7rYlwlG9AwdqjcZW7cme9xQguGc\nW+2cWwt0nQUxC1jmnFtZ2G6Hc86JSCMw2Dm3sLDdT4Cqp6G1t+sJa2iodg/pZvBg7X+/caOFo6Jm\n2rR0JL7TnvCG6gVj2zZdBTNsu5PGRr/tQdIoGOAnLBVXDmMKgIg8JCKLRORzhfGjgdai7VoLY1Wx\ncaOKRd/Mz1fvmSCPYYIRLWnwMDo6YMGCdCe8oXrBiCpc3NBgHkZ3+Cit7VUwROSRQs4h+FlR+P2X\nJV7WFzgb+BvgHOAyETkvIpvfJM/hqIAgLGWCES1p8DBWr9ZZ52n3kMMIRhTfT58hqT174ODBdLYe\n8lFa2+u9uXNuZhX7bQV+75zbASAiDwDTgf8CiuezjgXaSu1ozpw5bz5ubm6muahlZFqVP0omT9Yq\nmpUr4aSTfFuTH4K5GM75ayuThXAUaGjUOb14VjLbOaoKRp+CEXhJaWw9NGmSFk0AtLS00NLSEvsx\nowzmFJ/SecDnRKQfcBB4J/DPzrlNIrJLRM4AFgJXAbeU2mmxYHSlFjyMSZPgX/9V14AYOtS3Nflh\n2DAYMkQrpcaP92PDk0+mPxwFerEMvIxKBCMPHkaab0onTYKf/1wfd72Znjt3bizHDFtWO1tE1gMz\ngPtF5EEA59xO4P8Bi4AlwCLn3EOFl10D3AasAdYWjVdMnudgBEyeDAsXWjgqDnznMbJQIRVQTVgq\nKsEYOVL7ObW3975t1KRdMFKXwyiFc+4e59w451x/51yTc+7ioufucM6d4Jw7yTl3fdH4Yufcic65\nY51znwxz/DzPwQiYPFl/m2BEj888xvbtejE64QQ/x68Un4JRV6cNN7dsCb+vSkmzYIwbd3hH67jJ\n9EzvWghJNTZqWaIJRvT49DAWLIDTT89OhZ9PwQB/Yak0C0bQ0fqVV5I7ZqYFoxZCUiJw5ZXZCV1k\nCZ9NCLMUjgITjLSSdFgqs4IR1SzSLPCDH+gXxoiW4kqppFmwAGbMSP641VKpYLz+OuzdG93aLSYY\n3ZP0XIzMCsa2bRqqCTuL1KhdRozQBalaW3vfNkqcg8WL4bTTkj1uGCoVjA0b9DVRLYNrgtE9Sc/F\nyKxgpP0faWQDH63OX3pJW76kfcJeMZUKRtT5RR+CEbWXFAcWkiqTWkh4G/HjI4+xaBGcemqyxwyL\nb8FoaEieNg5eAAARdElEQVS+n1TwHtI4aS/ABKNMaiHhbcSPDw9j0aJshaNA50Ls2wdvvFHe9lGX\nvPvwMLIQxTDBKJNamINhxI8vDyNrgiFSWRPAPISksiAYI0ZoE8sdO5I5XqYFwzwMIyxJV0p1dGjC\nO2shKagsLBV1BMAEo3tEkvUyMisYFpIyomDkSF2zoa1kC8zoeP55vStMcyK1JyoRjKhv6IYOhf37\nNRGdFFkQDEi2tDazgmEhKSMqksxjZDEcFeBTMIKQWJKJ76wIRpKltZkWDPMwjChIMo9RC4LR0aHb\njRkT7fGTDktlSTDMwyhBUK0xYoRvS4w8YB5GeZQrGFu2aOv4fv2iPb4JRveYYPRCFuqjjeyQlIdx\n6BAsXQrTp8d/rDgoVzDi8v6TFIwDB7Sj8OjRyRwvDCYYvWAJbyNKAg8j7kqpNWv0AjR8eLzHiYta\nEoygtUldXTLHC8OECboQ2MGD8R8rk4JhCW8jSkaO1PDJhg3xHifL4SioLcHISjgK4MgjtSAgiZ5o\nmRUM8zCMKEliMaWsC8bo0dr0s7c72bi+n0lWSWVJMCC50tpMCoaFpIyoSWIxpawLRt++6o1t3lx6\nO/MwkiepPEYmBcNCUkbUxO1hHDwIy5ZlN+EdUE5YygQjeZKai5FZwTAPw4iSuD2M557TC9CQIfEd\nIwnKEYy4LrZBL6sk2rhkUTDMw+iBrP0zjfQTeBhxXYyyHo4K8OlhDByo61jv3h39vruStWuMCUYP\ntLfD1q22ZKkRLUcdpRejStetLpdaEYx9+7TnU1yTaivpmBsGE4zuyZxgbNqkX+6+fX1bYuSNOCfw\n1YpgtLVpS5C4JtU2NsZfKXXwoCb2s3RTOno0vPZa/N5X5gTDEt5GXMTVIqS9HVauhFNOiX7fSVOO\nYMSZX0wi8b1pE4wapR5nVgjanMed+M6kYFjC24iDuDyMVatg4kRdxzvr1IJgZC0cFZDEXIzMCYbN\nwTDiIi4PIy/hKDDBSDPmYXSDhaSMuIhr9b08CUaQQ+jpHJlg+COJxHcmBcM8DCMORo/WZnNRX5Dy\nJBj9+ml567Zt3T8f9/cziSopE4yeyZxgZPWfaWSDqPMY+/fr/k4+Obp9+qZUWCoJDyPuKqmsXmNM\nMLrBPAwjTqLOY6xYAcceCwMGRLdP3/gWDPMwumfiRHj5ZV3xMC4yJRjOmWAY8RK1h5GncFRAT4Jx\n6JDe/Ue9NGsxo0frin5xXhSzKhgDBuiEyTjb9GdKMLZv74yhGkYcRO1h1JJgvPqqLg5VXx/fsevr\ntR9XTzmUsMS1HnlSxF1amynBMO/CiJugCWFUlVKLF8Opp0azr7TQk2Ak9f2MM/G9eTMMHaqLEmWR\nuPMYmRIMm4NhxM3o0dCnTzSJ1ddfh9Wr4aSTwu8rTfgWjDgT31kNRwXEPRcjU4JhczCMuBGJLo+x\nfDkcd5yGUfNEGgQjLg8jD4JhHkYBC0kZSRBVHiOP+QswwUgzqRYMEblcRFaKyCERmV40fqSI3CEi\ny0VklYhcV/Tc9ML4GhG5uZLjZf2faWSDqDyMvAtG1zyPCYZ/Ui0YwArgMuDxLuNXADjnTgJOAz4s\nIuMLz30fuNo5NwWYIiLvKvdg5mEYSWAeRmkGD9bQ3Z49bx03wfBPUxNMnRrf/kMJhnNutXNuLdC1\n+/0mYKCI1AEDgP3AbhFpBAY75xYWtvsJMLvc45lgGEkQRaXUvn3wwgtwwgnR2ZUmugtL5aFKKuuC\n0acP/O53Me4/jp065+YBu4GNwMvAd5xzO4GjgdaiTVsLY2WR9X+mkQ0aGvT35s3V72PpUhWLOOck\n+MSnYFiVlD96XbdORB4BGoqHAAd80Tn36x5e8z6gP9AIjAT+ICKPVmPgnDlzAF2EZu/eZkaObK5m\nN4ZRNsWVUg0NvW/fHXkNRwV0FYw9e3SlumHD4j92XCEp57Jbut/S0kJLS0vsx+lVMJxzM6vY79nA\nr5xzHcAWEfkjmst4AhhXtN1YoK3UjgLBWLsW7rwzvqUfDaOYadNUMM47r7rXL1oEzc2RmpQqugpG\n4F0k8f0cORJ27tSbyChXxdu2TdtrZLGTRHNzM81FH7i5c+fGcpwoQ1LFH5U/ARcAiMhAYAbwnHNu\nE7BLRM4QEQGuAu4tZ+c2B8NIkiCPUS2LF9eWh5HknXldnS6hGiZk2B0WjuqdsGW1s0VkPSoI94vI\ng4WnfgjUi8gK4CngNudc8PW7BrgNWAOsdc49VM6xLOFtJEmY0to9e2DdOt1HXunJw0iKOMJSJhi9\n02tIqhTOuXuAe7oZ3w+8v4fXLAZOrPRY9s80kiRMae0zz2g7kCjDJWmjO8FI8vsZR6WUXWN6JzMz\nvc3DMJKksVGTuFu2VP7avCe8IR0eRtSVUiYYvWOCYRjdIFK9l2GCET8WkvJDZgTD/plG0lSbx6gF\nwRgxAl57TTvygglGrZAZwTAPw0iaajyMnTv1zvu44+KxKS2IvPWibYJRG2RCMA4e1HhlU5NvS4xa\nohoPY8kSOOUULf3MO0FYKsj1NDYmd+yok97BpD0TjNJkQjBefRWOOirfVSdG+qjGw6iFcFTAmDEq\nGJs2Jf/9jDrpvWuX9mEaMiS6feaRTAiGhaMMHzQ1wYEDlVVKLVqUvyVZeyLwMHx8P6MOSa1fb95F\nOWRCMMxVNHwQVEpVEpbK+wzvYnwKxtChsH+/Jt6j4Mc/hvPPj2ZfeSYTgmEehuGLSvIY27erNzJl\nSrw2pQWfghEk3aMIS23cCP/xH3D99eH3lXdMMAyjBJXkMRYvhunTNRZeCwSC4avDa1RhqW9+E666\nyq4x5ZCJj7aFpAxfVOJh1FLCG/x6GBBNpVRrK/z0p3Dddb1va2REMMzDMHxRiYdRy4Lh44YuipDU\nN74BV1+dbElwlsmEYGR1URMj+4wZo8nVrVt737bWBGPUKNixQzvzZjEktW4d/OIX8PnPR2dT3km9\nYDhnHobhj+LV90qxeTPs3g2TJydjVxqoq9P5Fy+9lE3B+NrX4CMfUeEzyiP1grFzp04IGjzYtyVG\nrVKOYCxerPMvam1FyKYmGDTIz4S3MILxwgvwq1/BZz4TrU15J9R6GElgCW/DN+XkMWppwl4xTU2w\nb5+fY4dJen/1q/Dxj2sTRaN8Ui8YFo4yfHP88XD//aW3WbwY3ve+ZOxJE01NnR1rk6ZaD2P1avjN\nb+D556O3Ke+kPiRlgmH4plwPo5YS3gFNTf6+nw0NWiXlXGWvmzsXPvUpnS1uVEbqPQwLSRm+Ofpo\nbUGxbRuMHHn48xs36l32xImJm+adSy/VGe4+GDhQ85u7d5d/8V+1Ch59FH74w3htyyvmYRhGL/RW\nKRX0j6q1hDfAGWfARRf5O36lYak5c+Czn7UimmoxwTCMMijVhLBWw1FpoBLBWLYMnngCrrkmXpvy\nTOoFw0JSRhoo5WGYYPijkkqpG26Aa6/VUJZRHakXDPMwjDTQU+LbORMMn5TbHmTRIli4ED784fht\nyjOpF4w9e3Q2qWH4pCcPo61NRcO8YD+UG5K64Qb4whegf//4bcozqReMMWNqp120kV7GjYO9e7V3\nUjHBhL1aTHingXIEY/58WLEC/u7vkrEpz6T+UmzhKCMNiMDUqYd7GRaO8ks5gvGVr8CXvgRHHpmM\nTXkm9YJhrr6RFrrLY9TSkqxppLek9+9/r32jPvjB5GzKM6kXDPMwjLTQNY9hCW//lPIwnFPv4stf\n1gl+RnhMMAyjTLp6GK+8oheiMWP82VTrjB6ta5V0dBz+3GOPwYYNcOWVyduVV1IvGBaSMtJCVw/D\nvAv/1Ndra/Vt29467px6FjfcAH1T3wApO6ReMMzDMNLC+PHat2jnTv3bBCMddBeWmjdPK9quuMKP\nTXkl9YJhHoaRFrpWSplgpIOughHkLubM0VUBjehIvWA0Nfm2wDA6CfIYQcK7FhdNShtdK6Xuvx/e\neAMuv9yfTXkl9YJRX+/bAsPoJMhjvPiiLk3a0ODbIqPYwwi8i7lzbcJvHNgpNYwKCDwMC0elh+J+\nUr/6lYYOZ8/2a1NeCSUYIvItEXlORJaKyP+IyJCi564XkbWF52cVjU8XkeUiskZEbg5zfMNImsDD\nsAl76SHwMDo6tCrqH//RWrXERVgP42FgmnPuFGAtcD2AiBwPvBeYClwM3Cry5r/w+8DVzrkpwBQR\neVdIG2qClpYW3yakBp/nYvx4rZJ69NF0CIZ9LjoFY86cFgYOhEsu8W1RfgklGM65R51zwZSZBUBQ\n0/Ru4OfOuYPOuZdRMTlDRBqBwc65hYXtfgKY81gGdmHoxOe56NNHK6WeeSYdCW/7XGgeqa0Nbr21\nxbyLmIkyh/Eh4IHC46OB9UXPtRXGjgZai8ZbC2OGkRmmTdP1u63tfjpobIQ//QkGDICZM31bk296\nnQMpIo8AxbUgAjjgi865Xxe2+SLQ7py7MxYrDSNFTJsG+/b5tsIIGDlSZ3Ofd555F3EjzrlwOxD5\nW+D/AOc75/YXxq4DnHPuxsLfDwE3AOuAx5xzUwvjVwDvdM59tId9hzPOMAyjRnHORS6fobqsiMhF\nwOeAcwOxKHAf8F8ichMacnob8LRzzonILhE5A1gIXAXc0tP+43jDhmEYRnWE8jBEZC1QDwStvxY4\n5z5WeO564GqgHfikc+7hwvipwO1AP+AB59wnqzbAMAzDSIzQISnDMAyjNkjlTG8RuUhE/lSY3Het\nb3viQETGisjvRGSViKwQkb8vjA8XkYdFZLWIzBORoUWvye1kSBHpIyJLROS+wt81eR4ARGSoiPyy\n8P5WiciZtXg+Cu9rVeE9/JeI1NfSeRCR20TkVRFZXjQW2fsvnM+fF14zX0TG92qUcy5VP6iIPQ9M\nAI4AlgLH+bYrhvfZCJxSeDwIWA0cB9wIfL4wfi3wzcLj44Fn0LzTxMI5CjzEp4DTC48fAN7l+/1V\ncT7+AfgZcF/h75o8DwXbbwc+WHjcFxhaa+ej8P1/Eagv/P0L4AO1dB6APwdOAZYXjUX2/oGPArcW\nHv9vdO5cSZvS6GGcAax1zq1zzrUDPwfe49mmyHHObXLOLS083gs8h058fA/w48JmP6ZzYmNuJ0OK\nyFjgL4B/LxquufMAUGivc45z7j8BCu9zF7V3PnYDB4CBItIX6I/O56qZ8+CcewLY0WU4yvdfvK+7\ngAt6symNgtF10l/uJ/eJyET0TmIB0OCcexVUVIDRhc3yPBnyJrTarjihVovnAeAYYKuI/GchRPdv\nIjKAGjsfzrkdwD8Dr6DvaZdz7lFq7Dx0w+gI3/+br3HOHQJ2isiIUgdPo2DUFCIyCFX3TxY8ja5V\nCLmuShCRS4BXC95WqTLqXJ+HIvoC04F/dc5NB/YB11F7n4tJaJhyAjAG9TTeR42dhzKI8v33Oo0h\njYLRBhQnX8YWxnJHwdW+C/ipc+7ewvCrItJQeL4R2FwYbwPGFb08OC89jWeFs4F3i8iLwJ3A+SLy\nU2BTjZ2HgFZgvXNuUeHv/0EFpNY+F6cBf3TObS/c/f4KeAe1dx66EuX7f/M5EakDhjjntpc6eBoF\nYyHwNhGZICL1wBXoRMA88h/As8657xaN3Qf8beHxB4B7i8avKFQ2HEPnZMhNwC4ROUNEBJ0MeS8Z\nwTn3BefceOfcJPR//Tvn3JXAr6mh8xBQCDesF5EphaELgFXU2OcCLQKZISL9CvZfADxL7Z0H4a13\n/lG+//sK+wD4a+B3vVrjuxKgh+qAi9APzFrgOt/2xPQezwYOoVVgzwBLCu97BPBo4f0/DAwres31\naPXDc8CsovFTgRWF8/Vd3+8txDl5J51VUrV8Hk5Gb5yWAnejVVI1dz7QvNYqYDmanD2ils4DcAew\nAdiP5nI+CAyP6v0DRwL/XRhfAEzszSabuGcYhmGURRpDUoZhGEYKMcEwDMMwysIEwzAMwygLEwzD\nMAyjLEwwDMMwjLIwwTAMwzDKwgTDMAzDKAsTDMMwDKMs/j8Ozzt93lGNTgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f94144ec9d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(iters,map(np.mean,session_rewards))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}