SimplyPPO replicates PPO with minimum (~250) lines of code in clean, readable PyTorch style, while trying to use as few additional tricks and hyper-parameters as possible.
- Advantage and state normalization.
- Gradient clipping.
- Entropy bonus.
- Tanh squashing to ensure action bounds and log_std clamping (as in SAC).
That's it! All other things follow the original paper.
Also check out SimplySAC, a minimal Soft-Actor-Critic PyTorch implementation.
Note
This is a single-threaded PPO implementation for continuous control tasks. The particular implementations of state normalization are adopted from here, where various other tricks are also discussed.
You can find the performance of Stable Baselines3 here as a reference.
These figures are produced with:
- One evaluation episode every 1e4 steps.
- 5 random seeds, where the mean return is represented by the solid line, and max/min return by the shaded area.
To execute a single run:
python learn.py -g [gpu_id] -e [env_id] -l [log_id]
Experiments use pybullet==3.0.8
.