InvertibleNetworks.jl

Building blocks for invertible neural networks in the Julia programming language.

Installation

] dev https://github.com/slimgroup/InvertibleNetworks.jl

Building blocks

• 1x1 Convolutions using Householder transformations (example)

• Residual block (example)

• Invertible coupling layer from Dinh et al. (2017) (example)

• Invertible hyperbolic layer from Lensink et al. (2019) (example)

• Invertible coupling layer from Putzky and Welling (2019) (example)

• Invertible recursive coupling layer HINT from Kruse et al. (2020) (example)

• Activation normalization (Kingma and Dhariwal, 2018) (example)

• Various activation functions (Sigmoid, ReLU, leaky ReLU, GaLU)

• Objective and misfit functions (mean squared error, log-likelihood)

• Dimensionality manipulation: squeeze/unsqueeze (column, patch, checkerboard), split/cat

• Squeeze/unsqueeze using the wavelet transform

Applications

• Invertible recurrent inference machines (Putzky and Welling, 2019) (generic example)

• Generative models with maximum likelihood via the change of variable formula (example)

• Glow: Generative flow with invertible 1x1 convolutions (Kingma and Dhariwal, 2018) (generic example, source)

To Do

• GPU support

Acknowledgments

This package uses functions from NNlib.jl, Flux.jl and Wavelets.jl

References

• Yann Dauphin, Angela Fan, Michael Auli and David Grangier, "Language modeling with gated convolutional networks", Proceedings of the 34th International Conference on Machine Learning, 2017. https://arxiv.org/pdf/1612.08083.pdf

• Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio, "Density estimation using Real NVP", International Conference on Learning Representations, 2017, https://arxiv.org/abs/1605.08803

• Diederik P. Kingma and Prafulla Dhariwal, "Glow: Generative Flow with Invertible 1x1 Convolutions", Conference on Neural Information Processing Systems, 2018. https://arxiv.org/abs/1807.03039

• Keegan Lensink, Eldad Haber and Bas Peters, "Fully Hyperbolic Convolutional Neural Networks", arXiv Computer Vision and Pattern Recognition, 2019. https://arxiv.org/abs/1905.10484

• Patrick Putzky and Max Welling, "Invert to learn to invert", Advances in Neural Information Processing Systems, 2019. https://arxiv.org/abs/1911.10914

• Jakob Kruse, Gianluca Detommaso, Robert Scheichl and Ullrich Köthe, "HINT: Hierarchical Invertible Neural Transport for Density Estimation and Bayesian Inference", arXiv Statistics and Machine Learning, 2020. https://arxiv.org/abs/1905.10687

Author

• Philipp Witte, Georgia Institute of Technology

• Contact at pwitte3@gatech.edu