How to consume of SQS queue in (massively)parallel fashion

[utkarshdeep]

Summary

Hi folks,

Let me begin by explaining my system design in short, post that will discuss the issue.

System Arch

I have an API(powered by AWS API Gateway & Lambda) via which client post multiple processing requests. This API takes the request, does some database operations & finally sends the processing request off to a SQS queue. At the other end, I have a consumer (NodeJs process managed using pm2) written using (sqs-consumer) which picks messages from the queue & processes it. I am using the pm2 in cluster mode on a 2 core machine.

Issue

The (sqs-consumer) processes messages one-at-a-time, hence under heavy load the messages on queue pile up leading to timeouts at client end. How can I process the message parallelly to handle the load?
Also is there a better way to handle this use cases like by using some other AWS service?

The API provides the option of batch processing using (handleMessageBatch), so please correct my understanding: using this we can consume at max 10 messages off the queue but eventually they will be processed one-by-one only?

Example

No response

[nicholasgriffintn]

Hey, sorry, we're not really here to provide AWS/ service support, we're just providing a package to use AWS services with.

I'll mark this as answered but keep it open if anyone wants to pop anything else in.