Skip to content

AWS Deployment Guide

Jump to bottom
Tristan Cole edited this page Jun 25, 2020 · 19 revisions

This is an overview of how to deploy the Sempo platform using Amazon Web Services (AWS). It utilises Amazon's Elastic Beanstalk for deploying Sempo inside docker-compose.

General Minimum Hardware Requirements:

A Burstable (t-class) Medium EC2 Instance
A deployment of the Sempo platform uses at least 3 docker containers.

  • The NGINX server
  • The main python app
  • The Ethereum worker (running two workers for high and low priority tasks is HIGHLY recommended) For loads of up to 5000 daily users, we've found a single medium instance running all containers is the most cost-effective and flexible approach.

A Burstable (t-class) Medium RDS Postgres Instance
Under most regular operation, Sempo only requires a small database. However during mass-disbursement operations and other high-use periods, the Ethereum Workers may open hundreds of connections to the database. During these periods a medium sized database server is required to prevent latency.

A Burstable (t-class) Small Elasticache Redis Instance
We use Redis as a message broker between works and the main python app. Load on the message broker tends not to be the main bottleneck, and so a small instance is normally sufficient.

Total Costs

  • AWS Costs: While costs will vary significantly depending on launch regions, and inclusion of other services such as load balancers and storage, a setup similar to the one above will cost APPROXIMATELY $1000 USD a month in aggregate.
  • Gas Costs: Setting up an account uses approximately 100k Gas and each transfer uses 50k Gas. As a scaling solution, we currently use the xDai side chain on the POA Network. So for 250k payments a quarter, the rough cost is $22 USD.

Set up an Elastic Container Registry to store your Docker Images

  1. Install the AWS CLI

  2. In the IAM Management section of AWS, create a user called ECR_BUILDER with programatic access only.

    • Attach the AmazonEC2ContainerRegistryPowerUser policy to the user.
    • Create and save the user's credentials to ~/.aws/credentials

  3. Create an Elastic Container Registry using the most up-to-date AWS Documentation

  4. Note down the repository URI, and set it as an environment variable called REPOSITORY_URI

  5. Note down the region that ECR is being used in, and set it as an environment variable called ECR_REGION

  6. Run ./devtools/prepare_aws_deploy.sh in to build and save your deployment docker images to AWS

Create an S3 bucket to store secrets:

  1. In the KMS console, create a new managed symmetric key named sempo-secrets
  2. In the S3 Management console, create a new bucket called sempo-secrets. Make sure the bucket has no public access
  3. In the IAM Management section of AWS, create a policy that provides read access to the secrets bucket. The following JSON policy will achieve this:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::ctp-prod-secrets"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:PutObjectAcl",
                "s3:ListObject"
            ],
            "Resource": "arn:aws:s3:::ctp-prod-secrets/*"
        }
    ]
}
  1. Create a new roll called EC2_WITH_SECRETS_ACCESS. The role should be copy of aws-elasticbeanstalk-ec2-role. This means an EC2 Service with Policies:
  • AWSElasticBeanstalkWebTier
  • AmazonS3ReadOnlyAccess
  • AWSElasticBeanstalkMulticontainerDocker
  • AmazonEC2ContainerServiceforEC2Role
  • AWSElasticBeanstalkWorkerTier Additionally, attach the policy you created in step 3
  1. Navigate back to the KMS console and attach the role as a user of the key you created

Create Application and Worker:

  1. Install the AWS EB CLI.

  2. In the root directory, run eb create.

    • Suggested name is blockchain-app-[Deployment_name]
    • Enter DNS CNAME prefix - Default
    • Choose application load balancer

  3. Confirm that the environment was created successfully (endpoint will not work yet).

  4. Navigate to the elastic beanstalk configuration page and under the "modify software" subsection, create an environment variable called DEPLOYMENT_NAME with value [Deployment_name], for example acme.

  5. In your config file set the APP_HOST to the url of the new instance, including http://

  6. Navigate to the EC2 console, select the instance and note down the instance id

  7. Note down the security group for the NON load balancer.

  8. For the instance select modify, and choose ‘Attach/replace IAM role’. Apply EC2_WITH_SECRETS_ACCESS

  9. In the EC2 sidebar, select load balancers, find the load balancer with the matching instance id. Note down the load balancer's endoing.

  10. Under listeners, choose edit and then add an https listener with a *.[your_domain] ssl cert. If you are NOT using AWS Route 53, you will first need to import your certificate

  11. Forward incoming traffic to the same target group used for http traffic.

  12. Add the https port to the load balancer security group.

  13. If you are using AWS Route 53, point an A record alias to the load-balancer endpoint. Otherwise, point a CNAME record at the endpoint.

Create SQL Database

  1. In RDS, launch a postgres instance

  2. Choose a medium for production (anything smaller will crash on bulk disbursements).

  3. Set the instance identifier to sempo-[deployment]

  4. Set a username to admin_[deployment]

  5. Set a password. Make sure you choose something with plenty of entropy, and don't forget it!

  6. Set the security group to the one noted in Create Application and Worker

  7. Set the database to be publicly accessible (we use this temporarily for setup)

  8. Launch

  9. Once launched, record down the endpoint name.

Set up Redis Message Broker

  1. Navigate to elasticache

  2. Select Create Cluster

  3. Select Redis

  4. Name sempo-[deployment]

  5. Add the instance to the security group noted down in Create Application and Worker

  6. Launch

  7. Once launched, record the primary endpoint

Set up security group

  1. With the security group noted in Create Application and Worker, note the group id
  2. Open up inbound ports for postgres (5432) redis (6379), http (80) with the source being the security group’s own id
  3. Open up an an inbound port for postres (5432) pointing to your local IP

Create database tables

  1. Log into the RDS Server using your chosen credentials (the host is the rds endpoint) and preferred interface (such as PGADMIN or PSQL)
  2. Create a database called app
  3. Create a database called eth_worker
  4. Set the RDS to no longer be publicly accessible
  5. Remove your IP from the security group

Create config and secrets files:

A Note on configs and secrets

Sempo uses three types of .ini files to configure the platform and store secrets.

  • common secrets files are used to store secrets that may be used across more than one deployment, such as keys for third-party services
  • specific secrets files are used to store secrets that change on a per-deployment basis, such as database passwords, and third-party project IDs.
  • specific config files are used to less sensitive settings that change on a per-deployment basis, such as contract blockchain addresses.

Templates for all three types of files can be found in /config_files/templates. The common secrets file should be named common_secrets.ini, while specific secrets and configs files should follow the following patterns:

  • Secrets: [deployment_name]_secrets.ini for example acme_secrets.ini.
  • Config: [deployment_name]_config.ini for example acme_config.ini.

The platform will run with limited functionality with most of common secrets empty. In order to quickly generate the necessary secrets to launch the platform, /config_files/generate_dev_test_secrets.py can be helpful, as it will create randomly seeded keys as required.

For reasonable general settings for specific config files, see /config_files/public/local_config.ini

For deployment

  1. For the specific config file, set:

    • [APP] deployment_name to the deployment name, for example acme
    • [DATABASE] host to the RDS endpoint name noted down
    • [DATABASE] database to app
    • [DATABASE] eth_database to eth_worker
    • [REDIS] uri to the primary redis endpoint recorded
    • [ETHEREUM] http_provider to an http RPC endpoint (you can easily obtain one from infura)
    • [ETHEREUM] contract_address to the address of the ERC20 token that you wish to use as the base reserve

  2. For the specific secrets file, set:

    • [DATABASE] password to the database password noted down
    • [DATABASE] user to postgres

  3. For the common secrets file, set the AWS ses_key_id and ses_secret to keys obtained for Amazon SES

  4. Upload the configs and secrets files to your secret bucket.

  5. For Secrets file, encrypt the file using the KMS key created in the 'Create an S3 Bucket With Secrets Access' section

Redeploy and set instance size

  1. In the AWS Elastic Beanstalk console for your app, select actions > Restart App Servers to apply the new configs and secrets
  2. After the app has rebooted, Navigate to the elastic beanstalk configuration page and under the "Capacity" subsection, set the EC2 instance type to t2.medium.
  3. The app should now function as expected!

To update your build:

  1. Run ./devtools/prepare_aws_deploy.sh again
  2. When this complete, run eb deploy

To use Circle CI as a deployment pipeline:

  1. In the AWS IAM Console, create a user called CircleCI with programmatic access only.
    • Attach the AWSElasticBeanstalkFullAccess policy to the user.
    • Note down the access key and its associated ID
  2. Create an account at Circle CI
  3. Add your Sempo Blockchain Repo as a project, choosing the manual setup options (we already have appropriate config files in the repo)
  4. Navigate to the "Environment Variables" section under "Project Settings"
  5. Create the following environment variables:
    • AWS_ACCESS_KEY_ID: The access key id for the CircleCI user
    • AWS_SECRET_ACCESS_KEY: The access key secret for the CircleCI user
    • DEPLOY_APPLICATION_NAME: The name of your Elastic Beanstalk Application
    • DEPLOY_ENV: The name of the specific Elastic Beanstalk Application environment to deploy to (eg staging or production)
    • DEPLOY_REGION: The Elastic Beanstalk Region
    • REPOSITORY_REGION: The Elastic Container Repository Region
    • REPOSITORY_URI: : The Elastic Container Repository URI
  6. Create a new branch: Circle CI Should now run on the branch. Note that only the master branch will deploy to elastic beanstalk
Clone this wiki locally