Distributed Training of Neural Networks

Implement popular distributed training algorithms for neural networks.

• Synchronous

  • SSGD Parallelized Stochastic Gradient Descent
  • MA Distributed Training Strategies for the Structured Perceptron
  • BMUF Block-wise Model Update Filtering
  • EASGD Deep learning with Elastic Averaging SGD

• Asynchronous

  • ASGD Distributed Delayed Stochastic Optimization
  • DCASGD Asynchronous Stochastic Gradient Descent with Delay Compensation

Requirements

• Deep learning library tinynn
• Distributed training framework Ray

Get started

# git clone
git clone https://github.com/borgwang/distributed-ml.git

cd distributed-ml
pip install -r requirements.txt

# sysc training
python distributed-ml/sync_train.py --algo SSGD
# asysc training
python distributed-ml/async_train.py --algo ASGD