This project serves as blueprint to get your python code running in AWS ECS Fargate, monitored and managed by CloudReactor. See a summary of the benefits of these technologies. This project is designed with best practices and smart defaults in mind, but also to be customizable.
It has these features built-in:
- Runs, tests, and deploys everything with Docker, no local python installation required
- Deploys to AWS ECS Fargate. Tasks can be scheduled, used as services, or executed only on demand.
- Sets up Tasks to be monitored and managed by CloudReactor
- Uses pip-tools to manage only top-level python library dependencies
- Uses pytest (automated tests), pylint (static code analysis), mypy (static type checking), and pip-audit (security vulnerability checking) for quality control
- Uses GitHub Actions for Continuous Integration (CI) and Continuous Deployment (CD)
This project deploys tasks by doing the following:
- Build the Docker image and send it to AWS ECR
- Create an ECS Task Definition and installs it in ECS
- Create or update a CloudReactor Task that is linked to the ECS Task Definition, so that it can manage it
The deployment method uses the aws-ecs-cloudreactor-deployer Docker image to build and deploy your tasks. (This is not to be confused with the Docker container that actually runs your tasks.) The deployer Docker image has all the dependencies (python, ansible, aws-cli etc.) built-in, so you don't need to install anything directly on your machine.
Sound good? OK, let's get started!
First, setup AWS and CloudReactor by following the pre-requisites. You'll be granting CloudReactor permission to start Tasks on your behalf, creating API keys, and optionally creating an IAM user/role that has permission to deploy your Tasks.
Next, you'll need to get this project's source code onto a filesystem where you can make changes. First fork the project, then clone your project:
git clone https://github.com/YourOrg/cloudreactor-python-ecs-quickstart.git
Afterwards, follow the remaining instructions starting from Set Task properties. You'll be setting the API keys and AWS credentials, optionally in a secure way using Secrets Manager. Finally, you'll deploy the Tasks with the command
./cr_deploy.sh <environment>
or a wrapper script that calls cr_deploy.sh with some options.
This project is configured to use the deployer image as a GitHub Action. After forking the source code, you should set these secrets in your GitHub account:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- CLOUDREACTOR_DEPLOY_API_KEY
For other configuration properties, view or modify
push.yml
which configures the GitHub Action.
You should change the aws-region to the region containing your ECS
Cluster. You may also want to change the
deployment-environment to the name of your deployment environment
(it defaults to staging). If you are running a private copy of the
CloudReactor API server, set the CLOUDREACTOR_API_BASE_URL secret value to
the base URL of your server (without the trailing slash).
Successfully deploying this example project will create two ECS tasks which are
listed in deploy_config/common.yml. They have the following behavior:
-
task_1 also prints 30 numbers and exits successfully. While it does so, it updates the successful count and the last status message that is shown in CloudReactor, using the status updater library. It is scheduled to run daily.
-
file_io uses non-persistent file storage to write and read numbers
-
web_server uses a python library dependency (Flask) to implement a web server and shows how to link an AWS Application Load Balancer (ALB) to a service. It requires that an ALB and target group be setup already, so it is not enabled by default. If enabling, you should also uncomment this line in the Dockerfile to allow the container to receive inbound requests:
EXPOSE 7070
The tasks are setup to be run with Docker Compose in docker-compose.yml. For example,
you can build the Docker image that runs the tasks by typing:
docker compose build
(You only need to run this again when you change the dependencies required by the project.)
Then to run, say task_1, type:
docker compose run --rm task_1
Docker Compose is setup so that changes in the environment file deploy_config/env/.env.dev
and the files in src will be available without rebuilding the image.
Now that you have deployed the example tasks, you can move your existing code to
this project. You can add or modify tasks in deploy_config/common.yml to call
the commands you want, with configuration for the schedule, retry parameters,
and environment variables.
Feel free to delete the tasks that you don't need, just by removing the top
level keys in task_name_to_config.
See the development guide for instructions on how to debug, add dependencies, and run tests and checks.
- Additional configuration options can be set or overridden
- If you want to be alerted when Task Executions fail, setup an Alert Method
- To avoid leaking secrets (passwords, API keys, etc.), see the guide on secret management
- For more secure networking, run your tasks on private subnets and/or tighten your security groups.
- If you're having problems, see the troubleshooting guide
Hopefully, this example project has helped you get up and running with ECS and CloudReactor. Feel free to reach out to us at [email protected] if you have any questions or issues!