HyP3

A processing environment for HyP3 Plugins in AWS.

Table of contents

• Developer Setup
• Deployment
  • Prerequisites
  • Stack Parameters
  • Deploy with CloudFormation
• Running the API Locally

Developer Setup

1. Clone the repository

   git clone [email protected]:ASFHyP3/hyp3.git
   cd hyp3
   

2. Create and activate a conda environment

   conda env create -f environment.yml
   conda activate hyp3
   

3. Run the tests

   make tests
   

Deployment

The deployment steps below describe how to deploy to a general AWS account you own/administer. HyP3 does support deploying to more secure environments which may require additional steps and are described in docs/deployments.

Prerequisites

These resources are required for a successful deployment, but managed separately:

• HyP3 plugin container images and tags. Current plugins are defined in job_spec.
• S3 bucket for CloudFormation deployment artifacts
• EarthData Login account authorized to download data from ASF
• IAM role configured for REST API access logging
• default VPC
• IAM user and roles for automated CloudFormation deployments (if desired)
• AWS Secrets Manager Secret containing key-value pairs for all secrets listed in the deployed job_specs
• For Earthdata Cloud deployments:
  • SSL certificate in AWS Certificate Manager for custom CloudFront domain name
  • ID of the CloudFront Origin Access Identity used to access data in S3
• For non-Earthdata Cloud deployments:
  • DNS record for custom API domain name
  • SSL certificate in AWS Certificate Manager for custom API domain name

Stack Parameters

Review the parameters in cloudformation.yml for deploy time configuration options.

Deploy with CloudFormation

To deploy HyP3 with reasonable defaults, follow the steps below. For more advanced deployment configuration, see the deployment GitHub Action.

From the repository root,

• Install dependencies for build and run

make install

• Install Python dependencies for AWS Lambda functions (requires pip for python 3.9)

make build

• Package the CloudFormation template

aws cloudformation package \
            --template-file apps/main-cf.yml \
            --s3-bucket <CloudFormation artifact bucket> \
            --output-template-file packaged.yml

• Deploy to AWS with CloudFormation

aws cloudformation deploy \
            --stack-name <name of your HyP3 Stack> \
            --template-file packaged.yml \
            --role-arn <arn for your deployment user/role> \
            --capabilities CAPABILITY_IAM \
            --parameter-overrides \
                "VpcId=<default vpc>" \
                "SubnetIds=<comma separated list of subnet ids>" \
                "EDLUsername=<EDL Username to download products>" \
                "EDLPassword=<EDL Password to download products>" \
                "DomainName=<Domain Name>" \
                "CertificateArn=<arn for ssl certificate>"

• Check API at https://<Domain Name>/ui

• (Optional) clean render and build artifacts

make clean

Note: this will remove any untracked files in the apps/ or lib/dynamo/ directory.

Running the API Locally

The API can be run locally to verify changes, but must be connected to a set of existing DynamoDB tables.

1. Set up AWS credentials in your environment as described here. Also see our wiki page.
2. Edit tests/cfg.env to specify the names of existing DynamoDB tables from a particular HyP3 deployment. Delete all of the AWS_* variables.
3. Run the API (replace <profile> with the AWS config profile that corresponds to the HyP3 deployment):

   AWS_PROFILE=<profile> make run

4. You should see something like Running on http://127.0.0.1:8080 in the output. Open the host URL in your browser. You should see the Swagger UI for the locally running API.
5. In Chrome or Chromium, from the Swagger UI tab, open Developer tools, select the Application tab, then select the host URL under Cookies. In Firefox, open Web Developer Tools, select the Storage tab, then select the host URL under Cookies. Add a new cookie with the following Name:

   asf-urs
   

   And the following Value:

   eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1cnMtdXNlci1pZCI6InVzZXIiLCJleHAiOjIxNTk1Mzc0OTYyLCJ1cnMtZ3JvdXBzIjpbeyJuYW1lIjoiYXV0aC1ncm91cCIsImFwcF91aWQiOiJhdXRoLXVpZCJ9XX0.hMtgDTqS5wxDPCzK9MlXB-3j6MAcGYeSZjGf4SYvq9Y
   

   And / for Path.
6. To verify access, query the GET /user endpoint and verify that the response includes "user_id": "user".