Skip to content

Spark Jobs and APIs

Jump to bottom
Awantik Das edited this page Mar 15, 2017 · 6 revisions

Execution process

  • Any Spark application spins off a single driver process (that can contain multiple jobs) on the master node that then directs executor processes (that contain multiple tasks) distributed to a number of worker nodes.
  • The driver process determines the number and the composition of the task processes directed to the executor nodes based on the graph generated for the given job. Note, that any worker node can execute tasks from a number of different jobs.

execution process

  • A Spark job is associated with a chain of object dependencies organized in a direct acyclic graph (DAG) such as the following example generated from the Spark UI. Given this, Spark can optimize the scheduling (for example, determine the number of tasks and workers required) and execution of these tasks.

dag

  1. Understanding Spark
  2. Spark Jobs & API
  3. Architecture
  4. RDD Internals
  5. Creating RDD
  6. Understanding Deployment & Program Behaviour
  7. RDD Transformation & Action
  8. Assignments 1
  9. Best Practices -1
  10. Introduction to DataFrame
  11. PySpark SQL
  12. Pandas to DataFrames
  13. Machine Learning with PySpark
  14. Transformers
  15. Estimators
  16. Spark Streaming
  17. Structured Streaming
  18. GraphX & GraphFrames
  19. Data Processing Architectures
  20. Problems
Clone this wiki locally