AWS Elastic Beanstalk Worker Environment

Hands-on architecture for worker-type applications with long-running tasks using Elastic Beanstalk. This configuration scales based on queue size and is highly-available across multiple availability zones.

Get to know Beanstalk Worker by reading the documentation.

Cloud Deployment

Use these instructions to create the infrastructure and deploy directly to the cloud.

First create the TF config file:

touch infra/.auto.tfvars

Set the environment according to your preferences:

region = "us-east-2"

availability_zone_a = "us-east-2a"
availability_zone_b = "us-east-2b"
availability_zone_c = "us-east-2c"

autoscaling_cooldown = 500
autoscaling_min_size = 1
autoscaling_max_size = 2

ec2_instance_types = "t2.micro"

sqs_daemon_max_concurrent_connections = 1
sqs_daemon_inactivity_timeout         = 499
sqs_daemon_visibility_timeout         = 500
sqs_daemon_max_retries                = 3

Create the infrastructure:

terraform -chdir="infra" init
terraform -chdir="infra" apply -auto-approve

Get the worker queue URL and set it as a variable to send messages soon:

aws sqs list-queues --queue-name-prefix awseb --region "<region>"

# copy the worker queue (not the dead letter)
export queue="https://sqs.<region>.amazonaws.com/<account>/<name>"

Enter the application directory:

cd worker

Get the dependencies and build it. Then init the EB application using the EB CLI and deploy:

npm i
npm run build

eb init
eb deploy

The required parameters are already set by Terraform.

Test it by sending messages to the queue:

# This will send 3 batches of 10 messages each = 30 messages in total
bash scripts/send-sqs-batch.sh 3

If everything worked accordingly the worker nodes should start processing the messages.

💡 CloudWatch alarms will trigger auto-scaling in or out when queue size meets the threshold

You may want to recreate the DynamoDB table between tests:

# Deletion process is asynchronous
bash scripts/delete-dynamodb-table

# Once deleted, create it again
bash scripts/create-dynamodb-table

Application Development

While in the application directory worker, create the .env:

touch .env

Add values according to your preferences:

LONG_RUNNING_TASK_DURATION=0
PORT=8080
DYNAMODB_REGION="us-east-2"
DYNAMODB_TABLE_NAME="BeanstalkTasks"

Install dependencies and start the application:

npm i
npm run dev

Send POST requests to localhost:8080 with the body being the message payload.

---

Clean-up

When done, delete the resources:

terraform destroy

References

Blog: AWS Elastic Beanstalk Worker Auto Scaling By Queue Size

AWS: General options for all Beanstalk environments

AWS: Deploying an Express application to Elastic Beanstalk

AWS: Using the Elastic Beanstalk Node.js platform

AWS: Configuring the application process with a Procfile

AWS: Extending Elastic Beanstalk Linux platforms

SO: nginx timeout