Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
d3fa248
Remove files/content related to Ruby/Dropbox version of project
floehopper Dec 14, 2018
b8dfe11
Add basic package.json
floehopper Dec 14, 2018
0cbe3f0
Add typescript package
floehopper Dec 14, 2018
51ecb74
Add build script to compile index.ts to index.js
floehopper Dec 14, 2018
223aa99
Add aws-cdk package
floehopper Dec 14, 2018
53a7f9c
Define empty aws-cdk stack & app
floehopper Dec 14, 2018
eb61865
Add cdk.json & document how cdk commands can be executed
floehopper Dec 14, 2018
68540aa
Add default constructor to stack class
floehopper Dec 15, 2018
a7b25e0
Create an EventRule to fire at daily at 2am
floehopper Dec 15, 2018
2cabad2
Add @aws-cdk/aws-lambda package
floehopper Dec 15, 2018
fd3e64d
Add empty Ruby lambda function as target of scheduled rule
floehopper Dec 15, 2018
0ea4cd7
Add empty Gemfile, constrain Ruby version & vendor gems
floehopper Dec 15, 2018
0c33a8c
Extract build.sh for individual lambda function
floehopper Dec 15, 2018
182f3f8
Make lambda function build scripts fail if any command fails
floehopper Dec 15, 2018
c78db03
Make dotenv gem available for enumerateTrelloBoards lambda function
floehopper Dec 15, 2018
1e42cf7
Document obtaining Trello API key & token and print in lambda function
floehopper Dec 15, 2018
52b61f1
Use trello gem to print name of first board
floehopper Dec 15, 2018
23f6a40
Add @aws-cdk/aws-sns package
floehopper Dec 15, 2018
03513a1
Create SNS Topic and publish message to it for each Trello board
floehopper Dec 15, 2018
1a5b273
Subscribe 2nd lambda function to SNS Topic and print events
floehopper Dec 15, 2018
85b71c9
Make dotenv gem available for backupTrelloBoard lambda function
floehopper Dec 15, 2018
25701aa
Download data for Trello board in backupTrelloBoard lambda function
floehopper Dec 15, 2018
02f10ab
Create a versioned S3 bucket & allow backupTrelloBoard to write to it
floehopper Dec 15, 2018
2e6e8c9
Write Trello board data to S3 bucket
floehopper Dec 15, 2018
03b02ea
Rename local constants which reference Lambda functions
floehopper Dec 23, 2018
8b30f73
Extract creating AWS resources into functions
floehopper Dec 23, 2018
36a3f96
Move granting of permissions into create functions
floehopper Dec 23, 2018
7bee932
Extract function to schedule lamda function invocation
floehopper Dec 23, 2018
967a288
Allow environment variables to be set from .env files
floehopper Dec 23, 2018
8d0ded5
Use env var to specify S3 bucket name for backups
floehopper Dec 23, 2018
4ec16c2
Email notification for errors in lamda functions
floehopper Dec 23, 2018
ab877f4
Extract reportErrors method to reduce duplication
floehopper Dec 23, 2018
9705993
Extract schedule expression into env var
floehopper Dec 23, 2018
19e9257
Move bucket name env var up into constructor
floehopper Dec 23, 2018
ae0e431
Extract Ruby Lambda runtime into constant to reduce duplication
floehopper Dec 23, 2018
b1ca055
Extract lambda function timeout into constant to reduce duplication
floehopper Dec 23, 2018
34f7e65
Install bundled gems from scratch
floehopper Dec 23, 2018
9693ccd
Avoid using ruby-trello gem to reduce dependencies
floehopper Dec 23, 2018
ff86d0c
Use Net::HTTP vs addressable & rest-client gems to reduce dependencies
floehopper Dec 23, 2018
8b5a4e7
Use single .env file & copy it into lambda function directories
floehopper Dec 24, 2018
dd6e01b
Add missing labels parameter when downloading board JSON
floehopper Dec 24, 2018
41ccd14
Re-order board query parameters
floehopper Dec 24, 2018
9f20085
Restrict fields returned from Trello API boards endpoint
floehopper Dec 28, 2018
e218ae4
Avoid duplication in S3 bucket name env var
floehopper Dec 29, 2018
a7c196f
Rename S3 bucket name env var to include "S3"
floehopper Dec 29, 2018
1c1a0ff
Rename alarm topic to monitoring topic
floehopper Dec 29, 2018
d552bf6
Make it clear when SNS subscription confirmation is sent
floehopper Dec 29, 2018
b63e324
Fix Markdown in README
floehopper Dec 29, 2018
df0d545
Rename backupTrelloBoardTopic -> backupBoardTopic
floehopper Dec 29, 2018
5145032
Rename enumerateTrelloBoardsFunction -> enumerateBoardsFunction
floehopper Dec 29, 2018
27b6139
Rename backupTrelloBoardFunction -> backupBoardFunction
floehopper Dec 29, 2018
5901917
Rename trelloBoardBackupsBucket -> boardBackupsBucket
floehopper Dec 29, 2018
ab8c11a
Change env var prefix to match project name
floehopper Dec 29, 2018
65cea17
Use project prefix even for dynamically generated env var
floehopper Dec 29, 2018
7564a40
Rename schedule expression env var
floehopper Dec 29, 2018
71327ee
Inline schedule function and use more specific naming
floehopper Dec 29, 2018
d634b0d
Add post-backup check
floehopper Dec 29, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
index.js
.env
23 changes: 0 additions & 23 deletions Capfile

This file was deleted.

14 changes: 0 additions & 14 deletions Gemfile

This file was deleted.

97 changes: 0 additions & 97 deletions Gemfile.lock

This file was deleted.

93 changes: 49 additions & 44 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,25 @@
We use this script to regularly backup all our GFR Trello boards to Dropbox.
## Requirements

## Generating/storing the tokens you need to use this script
* node
* npm

### Getting the Trello API key
## Setup

### Install node packages

npm ci

### Configure AWS

* You need to specify the AWS credentials and default region to use when running
the `cdk` command-line tool in the same way you would configure the `aws`
command-line tool. See [Configuring the AWS CDK Toolkit][1] for details.

* I recommend using `aws configure` and/or `~/.aws/credentials` & `~/.aws/config`
to setup a named profile and then use the `--profile <aws-profile-name>` option
with the `cdk` command-line tool to select the relevant AWS profile.

### Getting a Trello API key

Ensure you're logged in as the GFR Admin user.

Expand All @@ -13,11 +30,12 @@ Ensure you're logged in as the GFR Admin user.
$ export TRELLO_KEY=`pbpaste`

# Store the Trello API key in .env
$ echo "TRELLO_KEY=$TRELLO_KEY" >> .env
$ echo "TRELLO_KEY=$TRELLO_KEY" >> lambdaFunctions/enumerateTrelloBoards/.env
$ echo "TRELLO_KEY=$TRELLO_KEY" >> lambdaFunctions/backupTrelloBoard/.env

### Getting a token to allow this app to read our Trello board
### Getting a Trello API token

Ensure you're logged in as the GFR Admin user. It's safe to run do this step multiple times as you'll get the same token back even if it's already been generated.
Ensure you're logged in as the GFR Admin user on https://trello.com. It's safe to run do this step multiple times as you'll get the same token back even if it's already been generated.

$ open "https://trello.com/1/connect?key=$TRELLO_KEY&name=gfr-trello-backup&expiration=never&response_type=token"
# Allow the gfr-trello-backup app to read our Trello account
Expand All @@ -29,54 +47,41 @@ Ensure you're logged in as the GFR Admin user. It's safe to run do this step mul

# Store the Trello token in .env
$ echo "TRELLO_TOKEN=$TRELLO_TOKEN" >> .env
$ echo "TRELLO_TOKEN=$TRELLO_TOKEN" >> .env

### Getting a token to allow this app to write to Dropbox

Ensure you're logged in as the GFR Admin user.

$ open "https://www.dropbox.com/developers/apps"

# Navigate to "Trello Backup"
# Click the "Generate" button in OAuth 2 > Generated access token
# Copy the generated access token

# Temporarily store the access token in an environment variable
$ export DROPBOX_ACCESS_TOKEN=`pbpaste`

# Store the Dropbox token in .env
$ echo "DROPBOX_ACCESS_TOKEN=$DROPBOX_ACCESS_TOKEN" >> .env

## Testing the script locally

Assuming you've followed the instructions above and got all the tokens stored in .env, this should be as simple as running `ruby backup.rb`. If it's worked successfully then you'll see a backup for today in https://www.dropbox.com/home/Apps/Trello%20Backup.
### Environment variables

## Deployment
In the top-level `.env` file, set the following environment variables:

### Deploying with recap
* `TRELLO_BACKUP_SCHEDULE_FOR_BACKUP` - specifies how often the backup is performed, e.g. `cron(0 2 * * ? *)` runs daily at 2am (see [Schedule Expressions for Rules][3] for details)
* `TRELLO_BACKUP_SCHEDULE_FOR_CHECK` - specifies how often the post-backup check is performed, e.g. `cron(30 2 * * ? *)` runs daily at 2.30am
* `TRELLO_BACKUP_OLDEST_ALLOWED_BACKUP_IN_SECONDS` - how old a backup is allowed to be when the post-backup check is performed, e.g. `1800` only allows backups to be 30 minutes old; older backups trigger an error
* `TRELLO_BACKUP_S3_BUCKET_NAME` - the name of the S3 bucket you want to be created and used to save backups
* `TRELLO_BACKUP_MONITORING_EMAIL_ADDRESS` - the email address where monitoring emails will be sent (an email will be sent on first deployment to ask you to confirm the subscription)

$ cap bootstrap
$ cap deploy:setup
$ cap deploy
Note: this `.env` file is copied into some/all of the `lambdaFunctions` folders at build time.

### Set the application user's shell to bash
## Build

root$ chsh -s /bin/bash trello_backup
npm run build

### Set the Trello and Dropbox environment variables
## Execute CDK commands

I'm going to assume you've already got these environment variables configured in your local .env.
cdk --profile <aws-profile-name> <cdk-command>

$ cap env:set TRELLO_KEY=`grep TRELLO_KEY .env | cut -d"=" -f2`
$ cap env:set TRELLO_TOKEN=`grep TRELLO_TOKEN .env | cut -d"=" -f2`
$ cap env:set DROPBOX_ACCESS_TOKEN=`grep DROPBOX_ACCESS_TOKEN .env | cut -d"=" -f2`
* See [Command-line Toolkit][2] for a list of commands and options to use with
the `cdk` command-line tool.

## Test the script by running it manually
* The `cdk.json` file defines an appropriate value for the `--app` option, so
there is no need to specify this unless you want to override that value.

# Print the command that's being run under cron
trello_backup$ crontab -l | grep -o "/bin/bash.*backup.rb.*"
* `node_modules/.bin` must be in your `PATH` environment variable in order to
be able to run the `cdk` command-line tool.

# Copy and paste the printed command to execute the backup script manually
# If everything has worked then you should see a new backup file in
# https://www.dropbox.com/home/Apps/Trello%20Backup when you're logged
# in as GFR's dropbox user.
* When a stack uses assets (e.g. via a call to `aws-lambda.Code.asset()`),
before using the `cdk` command-line tool to deploy to an AWS environment for
the first time, the environment must be bootstrapped using: `cdk bootstrap`.

[1]: https://awslabs.github.io/aws-cdk/getting-started.html#configuring-the-cdk-toolkit
[2]: https://awslabs.github.io/aws-cdk/tools.html#command-line-toolkit-cdk
[3]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
56 changes: 0 additions & 56 deletions backup.rb

This file was deleted.

3 changes: 3 additions & 0 deletions cdk.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"app": "node index.js"
}
6 changes: 0 additions & 6 deletions config/schedule.rb

This file was deleted.

Loading