MLCommons™ Algorithmic Efficiency

Installation • Rules • Contributing • License

---

MLCommons Algorithmic Efficiency is a benchmark and competition measuring neural network training speedups due to algorithmic improvements in both training algorithms and models. This repository holds the competition rules and the benchmark code to run it.

Installation

1. Create new environment, e.g. via conda or virtualenv:

   Python minimum requirement >= 3.7

    sudo apt-get install python3-venv
    python3 -m venv env
    source env/bin/activate

2. Clone this repository:

   git clone https://github.com/mlcommons/algorithmic-efficiency.git
   cd algorithmic-efficiency

3. Install the algorithmic_efficiency package:

   pip3 install -e .

   Depending on the framework you want to use (e.g. JAX or PyTorch) you need to install them as well. You could either do this manually or by adding the corresponding options:

   JAX (GPU)

   pip3 install -e .[jax-gpu] -f 'https://storage.googleapis.com/jax-releases/jax_releases.html'

   JAX (CPU)

   pip3 install -e .[jax-cpu]

   PyTorch

   pip3 install -e .[pytorch] -f 'https://download.pytorch.org/whl/torch_stable.html'

   Development

   To use the development tools such as pytest or pylint use the dev option:

   pip3 install -e .[dev]

Docker

Docker is the easiest way to enable PyTorch/JAX GPU support on Linux since only the NVIDIA® GPU driver is required on the host machine (the NVIDIA® CUDA® Toolkit does not need to be installed).

Docker requirements

• Install Docker on your local host machine.

• For GPU support on Linux, install NVIDIA Docker support.

  • Take note of your Docker version with docker -v. Versions earlier than 19.03 require nvidia-docker2 and the --runtime=nvidia flag. On versions including and after 19.03, you will use the nvidia-container-toolkit package and the --gpus all flag. Both options are documented on the page linked above.

Setup

1. Clone this repository:

   git clone https://github.com/mlcommons/algorithmic-efficiency.git

2. Build Docker

   cd algorithmic-efficiency/ && sudo docker build -t algorithmic-efficiency .

3. Run Docker

   sudo docker run --gpus all -it --rm -v $PWD:/home/ubuntu/algorithmic-efficiency --ipc=host algorithmic-efficiency

   Currently docker method installs both PyTorch and JAX

Running a workload

JAX

python3 algorithmic_efficiency/submission_runner.py --framework=jax --workload=mnist_jax --submission_path=baselines/mnist/mnist_jax/submission.py --tuning_search_space=baselines/mnist/tuning_search_space.json

PyTorch

python3 algorithmic_efficiency/submission_runner.py --framework=pytorch --workload=mnist_pytorch --submission_path=baselines/mnist/mnist_pytorch/submission.py --tuning_search_space=baselines/mnist/tuning_search_space.json

Rules

The rules for the MLCommons Algorithmic Efficency benchmark can be found in the seperate rules document. Suggestions, clarifications and questions can be raised via pull requests.

Contributing

If you are interested in contributing to the work of the working group, feel free to join the weekly meetings, open issues, and see the MLCommons contributing guidelines.