-
Notifications
You must be signed in to change notification settings - Fork 20
AWS Deployment Guide
This is an overview of how to deploy the Sempo platform using Amazon Web Services (AWS). It utilises Amazon's Elastic Beanstalk for deploying Sempo inside docker-compose.
A Burstable (t-class) Medium EC2 Instance
A deployment of the Sempo platform uses at least 3 docker containers.
- The NGINX server
- The main python app
- The Ethereum worker (running two workers for high and low priority tasks is HIGHLY recommended) For loads of up to 5000 daily users, we've found a single medium instance running all containers is the most cost-effective and flexible approach.
A Burstable (t-class) Medium RDS Postgres Instance
Under most regular operation, Sempo only requires a small database. However during mass-disbursement operations and other high-use periods, the Ethereum Workers may open hundreds of connections to the database. During these periods a medium sized database server is required to prevent latency.
A Burstable (t-class) Small Elasticache Redis Instance
We use Redis as a message broker between works and the main python app. Load on the message broker tends not to be the main bottleneck, and so a small instance is normally sufficient.
Total Costs
While costs will vary significantly depending on launch regions, and inclusion of other services such as load balancers and storage, a setup similar to the one above will cost APPROXIMATELY $1000 USD a month in aggregate.
-
Install the AWS CLI
-
In the IAM Management section of AWS, create a user called
ECR_BUILDER
with programatic access only.- Attach the
AmazonEC2ContainerRegistryPowerUser
policy to the user. - Create and save the user's credentials to
~/.aws/credentials
- Attach the
-
Create an Elastic Container Registry using the most up-to-date AWS Documentation
-
Note down the repository URI, and set it as an environment variable called
REPOSITORY_URI
-
Note down the region that ECR is being used in, and set it as an environment variable called
ECR_REGION
-
Run
./devtools/prepare_aws_deploy.sh
in to build and save your deployment docker images to AWS
- In the KMS console, create a new managed symmetric key named
sempo-secrets
- In the S3 Management console, create a new bucket called
sempo-secrets
. Make sure the bucket has no public access - In the IAM Management section of AWS, create a policy that provides read access to the secrets bucket. The following JSON policy will achieve this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::ctp-prod-secrets"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:PutObjectAcl",
"s3:ListObject"
],
"Resource": "arn:aws:s3:::ctp-prod-secrets/*"
}
]
}
- Create a new roll called
EC2_WITH_SECRETS_ACCESS
. The role should be copy of aws-elasticbeanstalk-ec2-role. This means an EC2 Service with Policies:
- AWSElasticBeanstalkWebTier
- AmazonS3ReadOnlyAccess
- AWSElasticBeanstalkMulticontainerDocker
- AmazonEC2ContainerServiceforEC2Role
- AWSElasticBeanstalkWorkerTier Additionally, attach the policy you created in step 3
- Navigate back to the KMS console and attach the role as a user of the key you created
-
Install the AWS EB CLI.
-
In the root directory, run
eb create
.- Suggested name is blockchain-app-[Deployment_name]
- Enter DNS CNAME prefix - Default
- Choose application load balancer
-
Confirm that the environment was created successfully (endpoint will not work yet).
-
Navigate to the elastic beanstalk configuration page and under the "modify software" subsection, create an environment variable called
DEPLOYMENT_NAME
with value[Deployment_name]
, for exampleacme
. -
In your config file set the APP_HOST to the url of the new instance, including http://
-
Navigate to the EC2 console, select the instance and note down the instance id
-
Note down the security group for the NON load balancer.
-
For the instance select modify, and choose ‘Attach/replace IAM role’. Apply
EC2_WITH_SECRETS_ACCESS
-
In the EC2 sidebar, select load balancers, find the load balancer with the matching instance id. Note down the load balancer's endoing.
-
Under listeners, choose edit and then add an https listener with a
*.[your_domain]
ssl cert. If you are NOT using AWS Route 53, you will first need to import your certificate -
Forward incoming traffic to the same target group used for http traffic.
-
Add the https port to the load balancer security group.
-
If you are using AWS Route 53, point an A record alias to the load-balancer endpoint. Otherwise, point a CNAME record at the endpoint.
-
In RDS, launch a postgres instance
-
Choose a medium for production (anything smaller will crash on bulk disbursements).
-
Set the instance identifier to sempo-[deployment]
-
Set a username to admin_[deployment]
-
Set a password. Make sure you choose something with plenty of entropy, and don't forget it!
-
Set the security group to the one noted in Create Application and Worker
-
Set the database to be publicly accessible (we use this temporarily for setup)
-
Launch
-
Once launched, record down the endpoint name.
-
Navigate to elasticache
-
Select Create Cluster
-
Select Redis
-
Name sempo-[deployment]
-
Add the instance to the security group noted down in Create Application and Worker
-
Launch
-
Once launched, record the primary endpoint
- With the security group noted in Create Application and Worker, note the group id
- Open up inbound ports for postgres (5432) redis (6379), http (80) with the source being the security group’s own id
- Open up an an inbound port for postres (5432) pointing to your local IP
- Log into the RDS Server using your chosen credentials (the host is the rds endpoint) and preferred interface (such as PGADMIN or PSQL)
- Create a database called
app
- Create a database called
eth_worker
- Set the RDS to no longer be publicly accessible
- Remove your IP from the security group
Sempo uses three types of .ini files to configure the platform and store secrets.
-
common secrets
files are used to store secrets that may be used across more than one deployment, such as keys for third-party services -
specific secrets
files are used to store secrets that change on a per-deployment basis, such as database passwords, and third-party project IDs. -
specific config
files are used to less sensitive settings that change on a per-deployment basis, such as contract blockchain addresses.
Templates for all three types of files can be found in /config_files/templates
.
The common secrets file should be named common_secrets.ini
, while specific secrets and configs files should follow the following patterns:
- Secrets:
[deployment_name]_secrets.ini
for exampleacme_secrets.ini
. - Config:
[deployment_name]_config.ini
for exampleacme_config.ini
.
The platform will run with limited functionality with most of common secrets
empty.
In order to quickly generate the necessary secrets to launch the platform, /config_files/generate_dev_test_secrets.py
can be helpful, as it will create randomly seeded keys as required.
For reasonable general settings for specific config
files, see /config_files/public/local_config.ini
-
For the
specific config
file, set:-
[APP] deployment_name
to the deployment name, for exampleacme
-
[DATABASE] host
to the RDS endpoint name noted down -
[DATABASE] database
toapp
-
[DATABASE] eth_database
toeth_worker
-
[REDIS] uri
to the primary redis endpoint recorded -
[ETHEREUM] http_provider
to an http RPC endpoint (you can easily obtain one from infura) -
[ETHEREUM] contract_address
to the address of the ERC20 token that you wish to use as the base reserve
-
-
For the
specific secrets
file, set:-
[DATABASE] password
to the database password noted down -
[DATABASE] user
topostgres
-
-
For the
common secrets
file, set the AWSses_key_id
andses_secret
to keys obtained for Amazon SES -
Upload the configs and secrets files to your secret bucket.
-
For Secrets file, encrypt the file using the KMS key created in the 'Create an S3 Bucket With Secrets Access' section
- In the AWS Elastic Beanstalk console for your app, select
actions > Restart App Servers
to apply the new configs and secrets - After the app has rebooted, Navigate to the elastic beanstalk configuration page and under the "Capacity" subsection, set the EC2 instance type to
t2.medium
. - The app should now function as expected!
- Run
./devtools/prepare_aws_deploy.sh
again - When this complete, run
eb deploy
- In the AWS IAM Console, create a user called
CircleCI
with programmatic access only.- Attach the
AWSElasticBeanstalkFullAccess
policy to the user. - Note down the access key and its associated ID
- Attach the
- Create an account at Circle CI
- Add your Sempo Blockchain Repo as a project, choosing the manual setup options (we already have appropriate config files in the repo)
- Navigate to the "Environment Variables" section under "Project Settings"
- Create the following environment variables:
-
AWS_ACCESS_KEY_ID
: The access key id for theCircleCI
user -
AWS_SECRET_ACCESS_KEY
: The access key secret for theCircleCI
user -
DEPLOY_APPLICATION_NAME
: The name of your Elastic Beanstalk Application -
DEPLOY_ENV
: The name of the specific Elastic Beanstalk Application environment to deploy to (eg staging or production) -
DEPLOY_REGION
: The Elastic Beanstalk Region -
REPOSITORY_REGION
: The Elastic Container Repository Region -
REPOSITORY_URI
: : The Elastic Container Repository URI
-
- Create a new branch: Circle CI Should now run on the branch. Note that only the
master
branch will deploy to elastic beanstalk