Skip to content

Internal workings of an RDD

Jump to bottom
Awantik Das edited this page Mar 17, 2017 · 1 revision

  • RDDs operate in parallel. This is the strongest advantage of working in Spark: Each transformation is executed in parallel for enormous increase in speed.

  • The transformations to the dataset are lazy. This means that any transformation is only executed when an action on a dataset is called. This helps Spark to optimize the execution.

  • For instance, consider the following very common steps that an analyst would normally do to get familiar with a dataset:

    • Count the occurrence of distinct values in a certain column.
    • Select those that start with an A.
    • Print the results to the screen.

  • As simple as the previously mentioned steps sound, if only items that start with the letter A are of interest, there is no point in counting distinct values for all the other items. Thus, instead of following the execution as outlined in the preceding points, Spark could only count the items that start with A, and then print the results to the screen.

  1. Understanding Spark
  2. Spark Jobs & API
  3. Architecture
  4. RDD Internals
  5. Creating RDD
  6. Understanding Deployment & Program Behaviour
  7. RDD Transformation & Action
  8. Assignments 1
  9. Best Practices -1
  10. Introduction to DataFrame
  11. PySpark SQL
  12. Pandas to DataFrames
  13. Machine Learning with PySpark
  14. Transformers
  15. Estimators
  16. Spark Streaming
  17. Structured Streaming
  18. GraphX & GraphFrames
  19. Data Processing Architectures
  20. Problems
Clone this wiki locally