Efficiently Download Large Files into AWS S3 with Step Functions and Lambda

This project demonstrates how to efficiently download large files from an SFTP server and upload them to AWS S3 using AWS Step Functions and Lambda functions.

Article

For a detailed explanation of the architecture and implementation, check out my Medium article: Efficiently Download Large Files into AWS S3 with Step Functions and Lambda.

Prerequisites

Make sure you have the following installed:

• AWS SAM CLI
• AWS CLI
• Python 3.12 or above

Setup

1. Clone this repository:

   git clone https://github.com/yourusername/efficient-s3-download-step-functions.git
   cd efficient-s3-dwnload-step-functions

2. Create virtual environment

   python -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt

3. Configure AWS credentials

   aws configure

4. Build the stack using SAM

   aws build
   aws deploy --guided

Conclusion

This project provides an efficient solution for downloading large files to AWS S3 using Step Functions and Lambda. Feel free to explore, contribute, and suggest improvements! Open pull requests and issues are welcome.