Data Engineering Project

Getting started

• Use venv
• source pyelt/envelt/bin/activate
• Add pathto/pyelt into elt/pyelt/envelt/lib/python3.11/site-packages/pyelt.pth

Objectives

• The purpose of elt is to automate a data pipeline.
  • Use an EC2 on AWS to run the project
  • Store raw data in a S3 bucket
  • Load data into an RDS instance
  • Display data in a Shiny App
• Structure a project using both R and Python.