terrarium.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b1370140",
   "metadata": {},
   "source": [
    "# Experiments in the Terrarium\n",
    "Simulating populations of primitive agents scavenging for resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3d3b0ed",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e37d7ca6",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install pygame"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39decd83",
   "metadata": {},
   "source": [
    "# If something doesn't work, this is the M1 nightly version of pytorch, which I use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "963d91ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10a01cce",
   "metadata": {},
   "source": [
    "# Experiment #1 - World.py\n",
    "First task: simulate actual evolution. Each agent has a neural network with a single hidden layer of size 12. Inputs: dy, dx of nearest food; outputs: direction to walk in. Yes, they're just evolving to output their inputs... not so easy for them as it sounds though.\n",
    "\n",
    "At the end of each generation, the top %25 agents stay into the next (this is referred to as \"elitism.\" The rest of the agents are mutated versions of the top agents.\n",
    "\n",
    "We can specify the mutation rate with --mutation_rate. I use a dynamic mutation rate: if the agents aren't performing well, the mutation rate will be close to the specified rate. If they are performing well, the mutation rate will be lower. The rate adjusts at every generation. Thus, it's okay to go for a slightly high mutation rate to speed up evolution. I'd start with 0.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e42c23f9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! python world.py --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87744d96",
   "metadata": {},
   "source": [
    "### Random agents\n",
    "To see how agents start off at generation 0, run this cell. The little green items are food."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "249c5336",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pygame 2.3.0 (SDL 2.24.2, Python 3.11.3)\r\n",
      "Hello from the pygame community. https://www.pygame.org/contribute.html\r\n"
     ]
    }
   ],
   "source": [
    "! python world.py --mode display"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d48250cf",
   "metadata": {},
   "source": [
    "### All initialized as best agent from my training (200 generations)\n",
    "**By the way, you can click on the screen to make food appear.**\n",
    "These have a continuous action space so they are very smooth compared to the next experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f8873a4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pygame 2.3.0 (SDL 2.24.2, Python 3.11.3)\n",
      "Hello from the pygame community. https://www.pygame.org/contribute.html\n",
      "Loading model...\n"
     ]
    }
   ],
   "source": [
    "! python world.py --mode display --model_path \"best_agent_world_demo.pt\" --num_foods 30"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e486454",
   "metadata": {},
   "source": [
    "### Training test\n",
    "If you want to try training them, execute this cell. It takes about 400 generations for them to start getting decently good. That takes about 20 minutes(?) Change it to a smaller number if you just want to test it.\n",
    "\n",
    "Note: this cell failed on me when running in this notebook. If it doesn't work, just paste it into a command line. It should work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5626a10c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! python world.py --mode train --save_path \"training_test_demo.pt\" --generations 400 --mutation_rate 0.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c719892d",
   "metadata": {},
   "source": [
    "# Experiment 2: Hive Brain Reinforcement Learning\n",
    "It's not really a hive brain, but kinda! All agents have copies of the same model. Every 10 steps, the model is updated with the experiences of all of the agents. They all learn collectively."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b395f656",
   "metadata": {},
   "source": [
    "## Easy task - just eat food\n",
    "This trains WAY faster than their genetic counterparts. I did this as a proof of concept to see if I could make the collective learning work. They are also very fun to watch.\n",
    "\n",
    "The demo model was trained for 10 epochs. Performance plateaus after."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b4d0c042",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pygame 2.3.0 (SDL 2.24.2, Python 3.11.3)\n",
      "Hello from the pygame community. https://www.pygame.org/contribute.html\n",
      "Loaded model from models/hive_easy_trained_10_ep.pt\n"
     ]
    }
   ],
   "source": [
    "! python hivebrain_easy.py --model_path \"models/hive_easy_trained_10_ep.pt\" --display --num_foods 100"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e13cff7",
   "metadata": {},
   "source": [
    "### Train it yourself\n",
    "From my trials it actually gets close to peak performance in about 5 epochs, which only takes a few minutes to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a76e8528",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! python hivebrain_easy.py --train --epochs 10 --steps 10000 --save_path \"hive_easy_test_demo.pt\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7975f144",
   "metadata": {},
   "source": [
    "# Difficult task - Cooperate to bring food to colony\n",
    "Now, instead of having food appear randomly on the map, there is only one big food location with infinite food. Agents can pass food to each other. They get rewarded for getting food to their colony.\n",
    "\n",
    "The goal here was to have them learn cooperative behavior: the food passing radius is fairly large, so the optimal policy would be cooperative. Every agent that participated in a given piece of food travelling to the colony gets rewarded.\n",
    "\n",
    "The real question is whether sharing the same brain here is a good idea."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c34fe65d",
   "metadata": {},
   "outputs": [],
   "source": [
    "! python hivebrain_coop_dumb.py --display --model_path \"models/hive_e30_s6.pt\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "evolution",
   "language": "python",
   "name": "evolution"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}