Lambda architecture

https://www.databricks.com/glossary/lambda-architecture

https://www.snowflake.com/guides/lambda-architecture/

https://medium.com/@rahul.singh.suny/lambda-and-kappa-architectures-on-databricks-a-comprehensive-guide-0343256a7d33

History

Innovated by BackType, which was acquired by Twitter in 2011.

• Its implementation is released as Apache Storm

Feats

• Reliability
• Scalability
• Popularity: 当前主流

Components

Batch Layer (Cold Path)

• For accurate analysis
• Fault-tolerant and rerun support due to the immutability

Speed Layer (Hot Path)

• Analyzes data in real-time
• Optimized for low latency by compromise on accuracy

Serving Layer merges results from both batch and speed layers

Drawback

• Duplicate transformation code
• 2 platforms (Batch and Speed) to manage, including reconciling discrepancies

Vendors

• Apache Storm