GCP Data Pipeline Deployment with Terraform

Initial Setup

• Install Terraform
• Install the gcloud CLI

From the root of this repository:

git submodule init

From the root of this repository:

cd dev
terraform init

From the root of this repository:

cd prod
terraform init

You will need read & write permissions to the climateiq-state and test-climateiq-state GCS buckets, as this is where Terraform state is stored.

Troubleshooting

invalid_grant or reauth related error (invalid_rapt) error

"invalid_grant" error in Terraform when trying to access your Cloud Storage backend usually points to an issue with your authentication credentials and permissions. To fix this, reauthenticate your Google Cloud SDK by running:

gcloud auth application-default login

This will open a browser window for you to log in and refresh your credentials.

Deployment

The code for the Cloud Functions deployed by this Terraform are in the climateiq-cnn and climateiq-frontend repositories. They are included in this Terraform repository as Git submodules. Before deploying, you need to get the most recent version of the code from those repositories. To do so:

git submodule update --remote --init --recursive

To deploy to staging, from the root of this repository:

cd dev
terraform apply

To deploy to production, perform the same steps but from the prod directory

cd prod
terraform apply

Cloud function testing

The following command is an example testing the cloud functions via HTTP request.

gcloud functions call generate-feature-matrix-http --region us-central1 --data '{ "bucket": "test-climateiq-study-area-chunks", "name": "Test_NYC_Heat/Test_Config_Group/met_em.d03.2015-05-22_00:00:00.npy" }'