Skip to content

Running the model with the Arena

License

Notifications You must be signed in to change notification settings

emma-heriot-watt/arena-evaluation

Repository files navigation

EMMA: Evaluation on the Alexa Arena

Python 3.9 PyTorch Poetry
pre-commit style: black wemake-python-styleguide

Continuous Integration Tests

Important

If you have questions or find bugs or anything, you can contact us in our organisation's discussion.

About

We use code in this repository to generate new missions to facilitate self-play, to run the agent in the environment, and to evaluate the agent for the eval.ai leaderboard.

Important

We only ever ran this on a Ubuntu 20 Linux machine. This has not been tested or verified on MacOS or Windows. Your mileage may vary.

Installing dependencies

You can run the convenience script at scripts/prepare-user-area.sh.

Caution

I HIGHLY RECOMMEND reading the script to know what it does, because you might not need all of it! The convenience script does do some sudo-based file changes so if you don't want to permanently dirty your computer, look at it!

Running things

Tip

If you need sudo to run Docker, prefix the poetry run command with: sudo -E env PATH=$PATH.

T1 evaluation (the one for evai.ai)

When running T1, progress is sent to wandb:emma-simbot/alexa-arena-evaluation. All session IDs are prefixed with T1-.

Steps

  1. Create a new tmux session.

  2. In one tmux pane, run sudo /usr/bin/X :1 &

  3. In the other pane, run:

    poetry run python -m simbot_offline_inference run-background-services
  4. Finally, in a third pane, run:

    poetry run python -m simbot_offline_inference run-their-evaluation
  5. Let it run.

Running the trajectory generation

When running trajectories, each one is a new "run", and all the runs are tracked at wandb:emma-simbot/arena-high-level-trajectories.

Steps

  1. Create a new tmux session.

  2. In one tmux pane, run sudo /usr/bin/X :1 &

  3. In the other pane, run:

    poetry run python -m simbot_offline_inference run-background-services
  4. Finally, in a third pane, run:

    poetry run python -m simbot_offline_inference generate-trajectories
  5. Let it run.

Contributors 4

  •  
  •  
  •  
  •  

Languages