The Docker swarm cluster with Apache Spark and HDFS filesystem to run SageCal. The setup uses Docker images to build scaleble cluster.
Name | Version |
---|---|
Hadoop | 2.9 |
Spark | 2.2.0 |
java | OpenJDK-8 |
SageCal | - |
Note: The setup was only tested on Linux64 system.
docker pull fdiblen/spark-worker-dirac docker pull fdiblen/spark-master-dirac docker pull fdiblen/hadoop
Intallation instructions INSTALL.md
Instructions to submit a job JOBS.md