Skip to content

PySpark SQL Module

Jump to bottom
Awantik Das edited this page Mar 24, 2017 · 2 revisions

Important classes of Spark SQL and DataFrames:

  • pyspark.sql.SQLContext Main entry point for DataFrame and SQL functionality.
  • pyspark.sql.DataFrame A distributed collection of data grouped into named columns.
  • pyspark.sql.Column A column expression in a DataFrame.
  • pyspark.sql.Row A row of data in a DataFrame.
  • pyspark.sql.GroupedData Aggregation methods, returned by DataFrame.groupBy().
  • pyspark.sql.DataFrameNaFunctions Methods for handling missing data (null values).
  • pyspark.sql.DataFrameStatFunctions Methods for statistics functionality.
  • pyspark.sql.functions List of built-in functions available for DataFrame.
  • pyspark.sql.types List of data types available.
  • pyspark.sql.Window For working with window functions.
  1. Understanding Spark
  2. Spark Jobs & API
  3. Architecture
  4. RDD Internals
  5. Creating RDD
  6. Understanding Deployment & Program Behaviour
  7. RDD Transformation & Action
  8. Assignments 1
  9. Best Practices -1
  10. Introduction to DataFrame
  11. PySpark SQL
  12. Pandas to DataFrames
  13. Machine Learning with PySpark
  14. Transformers
  15. Estimators
  16. Spark Streaming
  17. Structured Streaming
  18. GraphX & GraphFrames
  19. Data Processing Architectures
  20. Problems
Clone this wiki locally