Big-Data-Project

File Name and Decription :

1. project.ipynb : This file is divided in to five sections which contains the code for preprocessing, parallelising the data preprocessing in Spark using Google Cloud Dataproc, parallelising the measuring of different configurations using Spark for first three sections respectively. For further sections the preprocessed data in Tensorflow/Keras is used and different parallelisation approaches for multiple GPUs are tested. Also, cherrypicking and hybrid parallel training of convolutional networks based on two papers are discussed in the report.

2. report.pdf - Project report.

Sequence for running the codes :

1. Open and run the project.ipynb file. Google console will be appropriate to perform the tasks. Also make sure your google account have allocation of the GPUs.