New serverless pattern - Serverless Messaging Redrive #2543

zalilias · 2024-12-16T16:41:26Z

Description of changes:
A serverless AWS solution for processing and fixing malformed messages using SQS queues and Lambda functions. This system automatically handles message validation, applies fixes where possible, and routes messages to appropriate queues based on their fixability. With built-in error handling and detailed logging, it provides a robust framework for message processing that can be easily extended for specific business needs.

The architecture uses AWS Lambda for processing, multiple SQS queues for message routing, and includes a dead-letter queue (DLQ) for messages requiring human intervention. Built with AWS SAM.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

benjymoses

You need to include a README.md with a detailed description of what this sample is, as well as deployment, testing, and deletion instructions.

You also need to include the *pattern.json file as per the sample adapted to your needs.

If any of the feedback / comments aren't clear, feel free to DM me offline to discuss and I can help point you in the right direction.

benjymoses · 2024-12-16T17:34:48Z

serverless-message-processing/functions/decision_maker/app.py

+import logging
+
+# Set up logging
+logger = logging.getLogger()


Is there an opportunity to use the PowerTools logging module here instead of Python native?

benjymoses · 2024-12-16T17:37:08Z

serverless-message-processing/functions/decision_maker/app.py

+
+def can_fix_message(message):
+    """
+    Determine if a message can be fixed automatically.


Can you include a sample scenario for customers to reference as a model? That could be "bad character replacement", "downstream system temporarily unavailable", or anything else you choose.

benjymoses · 2024-12-16T17:41:15Z

serverless-message-processing/functions/decision_maker/app.py

+MAIN_QUEUE_URL = os.environ['MAIN_QUEUE_URL']
+FATAL_DLQ_URL = os.environ['FATAL_DLQ_URL']
+
+def can_fix_message(message):


If you're using a function to determine if a message can be fixed or not, then as part of its return you need to know which "fix strategy" to use, and this needs to be a parameter for your fix_message function too doesn't it so that it knows which fix to apply.

benjymoses · 2024-12-16T17:42:00Z

serverless-message-processing/functions/decision_maker/app.py

+
+def fix_message(message):
+    """
+    Apply fixes to the message.


Same as on the other function, the wording needs to be expanded to help people with zero context understand what's meant to be happening here.

benjymoses · 2024-12-16T17:42:53Z

serverless-message-processing/functions/decision_maker/app.py

+        logger.error(f"Validation error: {str(e)}")
+        return False
+
+def fix_message(message):


Function definition needs to be expanded to include a fix strategy that you fork to within the function depending on what type of "fix" is needed, whether that's character replacement, a delay, etc.

benjymoses · 2024-12-16T17:46:26Z

serverless-message-processing/functions/decision_maker/requirements.txt

@@ -0,0 +1,2 @@
+boto3==1.26.137
+jsonschema==4.17.3


Is this being used directly anywhere?

benjymoses · 2024-12-16T17:46:54Z

serverless-message-processing/functions/processor/app.py

@@ -0,0 +1,121 @@
+import json
+import logging


Could you use PowerTools logging implementation?

benjymoses · 2024-12-16T17:49:32Z

serverless-message-processing/functions/processor/requirements.txt

@@ -0,0 +1,2 @@
+boto3==1.26.137
+jsonschema==4.17.3


Is this being used directly?

benjymoses · 2024-12-16T17:51:35Z

serverless-message-processing/template.yaml

+      Runtime: python3.9
+      CodeUri: ./functions/processor/
+      MemorySize: 128
+      Timeout: 29  


Is this always going to be adequate as this scales out? Consider parameterising this value (for both functions) and including more comments about the when it's declared at the top of this YAML doc describing when people might want to increase it.

benjymoses · 2024-12-16T17:51:58Z

serverless-message-processing/template.yaml

+      Runtime: python3.9
+      CodeUri: ./functions/decision_maker/
+      MemorySize: 128
+      Timeout: 29


Same comment about parameterisation as above

benjymoses · 2024-12-16T17:56:11Z

@ellisms I've been helping @zalilias with this and have done a first cursory review. It's worth letting him parse through that and integrate / reject my feedback before you do your review so you're making good use of your time.

…ents

benjymoses

Hey @zalilias thanks for bearing with me doing this next pass for you. This has come a long way but before we send to @ellisms for review I think there's a few more bits to go through.

As usual, if any of my comments don't make sense or you're not sure, just ping me 👍

benjymoses · 2025-01-15T11:16:58Z

serverless-message-processing/example-pattern.json

@zalilias you need to rename the file to be in line with the name of your pattern and replace the contents of the template / example with information about your pattern.

benjymoses · 2025-01-15T11:18:25Z

serverless-message-processing/README.md

+- Extensible processing logic
+
+## API Reference
+# Send Message


Good idea to include a copy of the API contract, but It would be nice to see commands that users can use in their terminal as an example curl https://${endpoint}/message.... with the relevant params to send a sample message.

benjymoses · 2025-01-15T11:19:34Z

serverless-message-processing/README.md

+4. Failed messages route to DLQs
+5. Decision maker attempts an automated recovery
+
+## Deployment


Likely to come up in @ellisms review, but we typically include a statement on users being responsible for spinning up / tearing down / paying for resources in their own AWS accounts. We'd also typically talk about having the AWS CLI installed and configured as a pre-requisite to using SAM. Take a look at some other SAM examples to get a feel for the norm.

benjymoses · 2025-01-15T11:20:50Z

serverless-message-processing/README.md

+## Basic Flow
+1. Messages enter through API Gateway
+2. Main queue receives messages
+3. Processor Lambda handles messages


Would include that this Lambda function polls the main queue using Event Source Mappings (ESMs) and handles the messages. Include links to docs / blogs / posts where they can build understanding.

benjymoses · 2025-01-15T11:22:05Z

serverless-message-processing/README.md

+1. Messages enter through API Gateway
+2. Main queue receives messages
+3. Processor Lambda handles messages
+4. Failed messages route to DLQs


If you're going to use terse statements in a numbered or bulleted list, it would be good to include a more full explanation alongside them. In this context what's a "failed message"? See if you can go a level deeper to explain what's going on as it may be confusing to somebody unfamiliar with the pattern or certain service features, like DLQ.

benjymoses · 2025-01-15T11:33:22Z

serverless-message-processing/template.yaml

@@ -2,6 +2,68 @@ AWSTemplateFormatVersion: '2010-09-09'
 Transform: AWS::Serverless-2016-10-31
 Description: Serverless message processing architecture


This description doesn't match what you're actually doing - this isn's just about message processing, it's about attempted automated failure re-drive.

benjymoses · 2025-01-15T11:35:09Z

serverless-message-processing/template.yaml

+      - Performing CPU-intensive operations
+      - Seeing timeout errors or slow processing
+      - Handling concurrent message processing
+      Recommended ranges:


What does "heavy" mean to the user? Isn't this ultimately dependent on how computationally expensive the operation would be? Perhaps in the context of a demo pattern think instead of suggesting 2 values with the lower as the default given it's just a sample and not meant to be production grade.

There would be an expectation that if a customer wanted to fork this and use it themselves that they'd do their own due diligence on memory sizing for their use case. Likely using Lambda Power Tuning.

benjymoses · 2025-01-15T11:35:29Z

serverless-message-processing/template.yaml

+    MinValue: 128
+    MaxValue: 10240
+
+  DecisionMakerMemorySize:


Same comments as for ProcessorMemorySize

benjymoses · 2025-01-15T11:36:52Z

serverless-message-processing/template.yaml

+    MinValue: 1
+    MaxValue: 900
+
+  DecisionMakerTimeout:


This parameter has all of the same constraints as the one above for the same reasons (they're Lambda functions). I'd collapse in to one parameter called LambdaTimeout and make use of it for both functions. Make sure to update the first line of the description.

benjymoses · 2025-01-15T11:37:31Z

serverless-message-processing/template.yaml

@@ -115,8 +178,9 @@ Resources:
      Handler: app.lambda_handler
      Runtime: python3.9


Python 3.9 is 4 years old now, should we not be defaulting to the current LTS as this is a newly published asset? 3.12 seems like a reasonable default.

New serverless pattern - Serverless Messaging Redrive

5048d72

julianwood assigned ellisms and julianwood Dec 16, 2024

benjymoses suggested changes Dec 16, 2024

View reviewed changes

New serverless pattern - Serverless Messaging Redrive- added improvem…

49e0acf

…ents

benjymoses reviewed Jan 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New serverless pattern - Serverless Messaging Redrive #2543

New serverless pattern - Serverless Messaging Redrive #2543

zalilias commented Dec 16, 2024

benjymoses left a comment

benjymoses Dec 16, 2024

benjymoses Dec 16, 2024

benjymoses Dec 16, 2024

benjymoses Dec 16, 2024

benjymoses Dec 16, 2024

benjymoses Dec 16, 2024

benjymoses Dec 16, 2024

benjymoses Dec 16, 2024

benjymoses Dec 16, 2024

benjymoses Dec 16, 2024

benjymoses commented Dec 16, 2024

benjymoses left a comment

benjymoses Jan 15, 2025

benjymoses Jan 15, 2025

benjymoses Jan 15, 2025

benjymoses Jan 15, 2025

benjymoses Jan 15, 2025

benjymoses Jan 15, 2025

benjymoses Jan 15, 2025

benjymoses Jan 15, 2025

benjymoses Jan 15, 2025

benjymoses Jan 15, 2025

		@@ -2,6 +2,68 @@ AWSTemplateFormatVersion: '2010-09-09'
		Transform: AWS::Serverless-2016-10-31
		Description: Serverless message processing architecture

		@@ -115,8 +178,9 @@ Resources:
		Handler: app.lambda_handler
		Runtime: python3.9

		@@ -0,0 +1,2 @@
		boto3==1.26.137
		jsonschema==4.17.3

		@@ -0,0 +1,2 @@
		boto3==1.26.137
		jsonschema==4.17.3

New serverless pattern - Serverless Messaging Redrive #2543

Are you sure you want to change the base?

New serverless pattern - Serverless Messaging Redrive #2543

Conversation

zalilias commented Dec 16, 2024

benjymoses left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjymoses commented Dec 16, 2024

benjymoses left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment