Apache Spark is a very popular open-source analytics engine for large-scale data processing. When building Spark applications as part of a production-grade solution, developers need to take care of engineering aspects such as inner dev loop, testing, CI/CD, infra-as-code and observability.
In this work-in-progress sample we'll demonstrate an E2E Spark data pipeline and how to tackle the above-mentioned engineering fundamentals.
The ETL scenario is based on a time series data processing pipeline. The pipeline reads synthetic data from a message broker, processes it with PySpark and writes the enriched data to a Database. The key point to note here is that the data processing logic can be shared between the cloud and local versions through the common_lib Wheel.
The IoT Telemetry Simulator is hosted in Azure Container Instances. It sends generated data to a Kafka broker, exposed through Azure Event Hubs. The ETL workload is represented in a Databricks Job. This job is responsible for reading and enriching the data from sources and store the final output to an Azure SQL DB.
The pipeline begins with Azure IoT Device Telemetry Simulator sending synthetic Time Series data to a Confluent Community Kafka Server. A PySpark app then processes the Time Series, applies some metadata and writes the enriched results to a SQL DB hosted in SQL Server 2022 Linux container.
In the Cloud version, we provision all infrastructure with Terraform. Prior to running terraform apply you must ensure the wheel ./src/common_lib/dist/common_lib-*.whl exists locally by executing python3 -m build ./src/common_lib.
In the local version, we provision and orchestrate everything with Docker Compose.
Please use the docker compose tool instead of the older version docker-compose.
- To validate that the local E2E pipeline is working correctly, we can execute the script
smoke-test.sh. This script will send messages using the IoT Telemetry Simulator and then query the SQL DB to ensure the messages were processed correctly. - Unit tests are available for the
common_libWheel in PyTest. - Both type of tests are also executed in the CI pipeline.
GitHub Actions is used for CI/CD. The CI pipeline runs the tests and the CD pipeline deploys the Cloud infrastructure and the Spark Job.
The local version of the solution also deploys additional containers for Prometheus and Grafana. The Grafana dashboard below, relies on the Spark 3.0 metrics emitted in the Prometheus format.
GitHub Codespaces are supported through the VS Code Dev Containers. The minimum required machine type configuration is 4-core.