New pattern submission - SQS and S3 integration using Lambda

[sidd130]

Description

This pattern demonstrates the use of an SQS queue with a Lambda function and an S3 bucket.

Language:

Python

Framework

Terraform

Services from/to

SQS to Lambda to S3

Description

In this pattern, a standard SQS queue accepts JSON messages and a Lambda function is tied to the SQS queue using a event source mapping, allowing the Lambda to poll the SQS queue. Whenever a message arrives at the SQS queue, Lambda function is triggered synchronously. Once the message has been processed by the Lambda function, it writes the message as a JSON object to the S3 bucket.

To enable the above integration, the following permission set up has been implemented:

• SQS queue policy - Allows sqs:ReceiveMessage, sqs:GetQueueAttributes, and sqs:DeleteMessage permissions on the SQS queue to the Lambda function.
• S3 bucket policy - Allows s3:PutObject permission on the S3 bucket to the Lambda function.

The pattern consists of the following AWS resources:

• SQS queue - This is where messages will be sent
• Lambda function - This will process the messages and save it on the S3 bucket
• S3 bucket - This will be used to store the processed messages
• S3 bucket policy - Permissions for the S3 bucket
• SQS queue policy - Permissions to read from the queue
• Lambda execution role - Permissions to read from the queue and write to the S3 bucket

Deployment commands

Deployment instructions are present in the pattern README.md.

GitHub PR for template:

#2702

Payload example (e.g. Lambda event payload from source service).

N/A

Additional resources (optional: link and anchor text, up to 5 resources)

• Amazon Simple Queue Service (Amazon SQS)
• AWS Lambda
• Amazon S3

Author bio
Name: Siddharth Das
Photo URL:
Linkedin handle: https://www.linkedin.com/in/siddharth-das-a2578257/
Twitter handle: N/A
Description (up to 255 chars): Principal Software Engineer at Fidelity Investments.