This is a Java implementation of some of the algorithms for training deep neural networks. GPU support is provided via the OpenCL and Aparapi. The architecture is designed with modularity, extensibility and pluggability in mind.
I'm using the git-flow model. The most stable (but older) sources are available in the master branch, while the latest ones are in the develop branch.
If you want to use the previous Java 7 compatible version you can check out this release.
- Multilayer perceptron
- Restricted Boltzmann Machine
- Autoencoder
- Deep belief network
- Stacked autoencodeer
- Convolutional networks with max pooling, average poolng and stochastic pooling.
- Maxout networks (work-in-progress)
- Backpropagation - supports multilayer perceptrons, convolutional networks and dropout.
- Contrastive divergence and persistent contrastive divergence implemented using these and these guidelines.
- Greedy layer-wise training for deep networks - works for stacked autoencoders and DBNs, but supports any kind of training.
All the algorithms support GPU execution.
Out of the box supported datasets are MNIST, CIFAR-10/CIFAR-100 (experimental, not much testing), IRIS and XOR, but you can easily implement your own.
Experimental support of RGB image preprocessing operations - affine transformations, cropping, and color scaling (see Generaltest.java -> testImageInputProvider).
- Logistic
- Tanh
- Rectifiers
- Softplus
- Softmax
- Weighted sum
All the functions support GPU execution. They can be applied to all types of networks and all training algorithms. You can also implement new activations.
- Java 8.
- To build the project you need gradle or maven. If you don't use any of these you can go to the project folder and execute the gradlew console command, which will automatically setup gradle environment for you.
- I'm also uploading the latest jar file (with bundled dependencies and sources) here.
- Depending on your environment you might need to download the relevant aparapi .dll or .so file (located in the root of each archive) from here and add it's location to the system PATH variable. (This)[https://code.google.com/p/aparapi/wiki/DevelopersGuideLinux] is a guide on how to set up OpenCL in linux environment.
The samples are organized as unit tests. If you want see examples on various popular datasets you can go to nn-samples/src/test/java/com/github/neuralnetworks/samples/.
There are two projects:
- nn-core - contains the full implementation.
- nn-samples - contains implementations of popular datasets and
The software design is tiered, each tier depending on the previous ones.
This is the first "tier". Each network is defined by a list of layers. Each layer has a set of connections that link it to the other layers of the network, making the network a directed acyclic graph. This structure can accommodate simple feedforwad nets, but also more complex architectures like http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf. You can build your own specific network.
This tier is propagating data through the network. It takes advantage of it's graph structure. There are two main base components:
- LayerCalculator - propagates data through the graph. It receives target layer and input data clamped to a given layer (considered an input layer). It ensures that the data is propagated through the layers in the correct order and that all the connections in the graph are calculated. For example, during the feedforward phase of backpropagation the training data is clamped to the input layer and is propagated to the target layer (the output layer of the network). In the bp phase the output error derivative is clamped as "input" to the layer and the weights are updated using breadth-first graph traversal starting from the output layer. Essentially the role of the LayerCalculator is to provide the order, in which the network layers are calculated.
- ConnectionCalculator - base class for all neuron types (sigmoid, rectifiers, convolutional etc.). After the order of calculation of the layers is determined by the LayerCalculator, then the list of input connections for each layer is calculated by the ConnectionCalculator.
Most of the ConnectionCalculator implementations are optimized for GPU execution. Aparapi imposes some important restrictions on the code that can be executed on the GPU. The most significant are:
- only one-dimensional arrays (and variables) of primitive data types are allowed. It is not possible to use complex objects.
- only member-methods of the Aparapi Kernel class itself are allowed to be called from the GPU executable code.
Therefore before each GPU calculation all the data is converted to one-dim arrays and primitive type variables. Because of this all Aparapi neuron types are using either AparapiWeightedSum (for fully connected layers and weighted sum input functions), AparapiSubsampling2D (for subsampling layers) or AparapiConv2D (for convolutional layers). Most of the data is represented as one-dimensional array by default (for example Matrix).
All the trainers are using the Trainer base class. They are optimized to run on the GPU, but you can plug-in other implementations and new training algorithms. The training procedure has training and testing phases. Each Trainer receives parameters (for example learning rate, momentum, etc) via Properties (a HashMap). For the supported properties for each trainer please check the TrainerFactory class.
Input is provided to the neural network by the trainers via TrainingInputProvider interface. Each TrainingInputProvider provides training samples in the form of TrainingInputData (default implementation is TrainingInputDataImpl). The input can be modified by a list of modifiers - for example MeanInputFunction (for subtracting the mean value) and ScalingInputFunction (scaling within a range). Currently MnistInputProvider and IrisInputProvider are implemented.
Ivan Vasilev (ivanvasilev [at] gmail (dot) com)