Adaptive Weighting in Contextual Bandits

Models for paper Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Bandits.

Table of contents
Overview • Development Setup • Quickstart

Overview

Note: For any questions, please file an issue.

Adaptive experimental designs can dramatically improve efficiency in randomized trials. But adaptivity also makes offline policy inference challenging. In the paper Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Bandits, we propose a class of estimators that lead to asymptotically normal and consistent policy evaluation. This repo contains reproducible code for the results shown in the paper.

We organize the code into two directories:

• ./adaptive is a Python module for doing adaptive weighting developed in the paper.

• ./experiments contains python scripts to run experiments and make plots shown in the paper, including:

  • collecting contextual bandits data with a Thompson sampling agent;
  • doing off-line policy evaluation using collected data;
  • saving results and making plots.

Development setup

R and Python are required. We recommend creating the following conda environment for computation.

conda create --name aw_contextual python=3.7
conda activate aw_contextual
source install.sh

Quickstart with model

• To do adaptive weighting and reproduce results shown in the paper, please follow the instructions in ./experiments/README.md.
• For a quick start on one simulation using synthetic data of sample size 1000 , use

source activate aw_contextual
cd ./experiments/
python script_synthetic.py -T 1000 -s 1 -n test

Results will be saved in ./experiments/results/