Release 12.0.3 (#3107)

* merge CUMULUS-2557 * merge CUMULUS-2971 * merge CUMULUS-3021 * merge CUMULUS-3024 * 12.0.3 version bump * remove duplicate api user * Sort results in unit test before comparison * add ci stack * CUMULUS-3027 -- Tightly constrain typescript version due to knex typing (#3095) * Tightly constrain typescript verseion due to knex typing Unpinned dependency resulted in typescript 4.8 being installed, however knex (all versions) has an outstanding bug: knex/knex#5279 * Add CHANGELOG 🔔 * Update generate-ts-build-cache to always 'npm install' * Update CHANGELOG * Update CNM lambdas to utilize newer releases (#3094) Co-authored-by: Charles Huang <[email protected]> Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Charles Huang <[email protected]>
nasa · Oct 10, 2022 · 13f055a · 13f055a
1 parent a35accb
commit 13f055a
Show file tree

Hide file tree

Showing 135 changed files with 2,439 additions and 655 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,31 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ## Unreleased
 
+## [v12.0.3] 2022-10-03 [BACKPORT]
+
+### Fixed
+
+- **CUMULUS-3024**
+  - Update PUT /granules endpoint to operate consistently across datastores
+    (PostgreSQL, ElasticSearch, DynamoDB). Previously it was possible, given a
+    partial Granule payload to have different data in Dynamo/ElasticSearch and PostgreSQL
+  - Given a partial Granule object, the /granules update endpoint now operates
+    with behavior more consistent with a PATCH operation where fields not provided
+    in the payload will not be updated in the datastores.
+  - Granule translation (db/src/granules.ts) now supports removing null/undefined fields when converting from API to Postgres
+    granule formats.
+  - Update granule write logic: if a `null` files key is provided in an update payload (e.g. `files: null`),
+    an error will be thrown. `null` files were not previously supported and would throw potentially unclear errors. This makes the error clearer and more explicit.
+  - Update granule write logic: If an empty array is provided for the `files` key, all files will be removed in all datastores
+- **CUMULUS-2971**
+  - Updated `@cumulus/aws-client/S3ObjectStore` class to take string query parameters and
+    its methods `signGetObject` and `signHeadObject` to take parameter presignOptions
+- **CUMULUS-2557**
+  - Updated `@cumulus/aws-client/S3/moveObject` to handle zero byte files (0 byte files).
+- **CUMULUS-3021**
+  - Updated `@cumulus/api-client/collections` and `@cumulus/integration-tests/api` to encode
+    collection version in the URI path
+
 ## [v12.0.2] 2022-08-10 [BACKPORT]
 
 ### Notable Changes
@@ -15,6 +40,12 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
       `cumulus_id`. Previous logic removed files by matching `file.cumulus_id`
       to `granule.cumulus_id`.
 
+### Changed
+
+- **CUMULUS-3027**
+  - Pinned typescript to ~4.7.x to address typing incompatibility issues
+    discussed in https://github.com/knex/knex/pull/5279
+  - Update generate-ts-build-cache script to always install root project dependencies
 ## [v12.0.1] 2022-07-18
 
 - **CUMULUS-2995**
@@ -5932,7 +5963,8 @@ Note: There was an issue publishing 1.12.0. Upgrade to 1.12.1.
 
 ## [v1.0.0] - 2018-02-23
 
-[unreleased]: https://github.com/nasa/cumulus/compare/v12.0.2...HEAD
+[unreleased]: https://github.com/nasa/cumulus/compare/v12.0.3...HEAD
+[v12.0.3]: https://github.com/nasa/cumulus/compare/v12.0.2...v12.0.3
 [v12.0.2]: https://github.com/nasa/cumulus/compare/v12.0.1...v12.0.2
 [v12.0.1]: https://github.com/nasa/cumulus/compare/v12.0.0...v12.0.1
 [v12.0.0]: https://github.com/nasa/cumulus/compare/v11.1.0...v12.0.0

diff --git a/bamboo/generate-ts-build-cache.sh b/bamboo/generate-ts-build-cache.sh
@@ -27,9 +27,10 @@ if [[ $USE_CACHED_BOOTSTRAP == true ]]; then
   git checkout "$GIT_SHA"
 else
   CURRENT_WORKING_DIR=$(pwd)
-  npm install
 fi
 
+npm install
+
 # Bootstrap to install/link packages
 npm run ci:bootstrap-no-scripts
 

diff --git a/bamboo/select-stack.js b/bamboo/select-stack.js
@@ -15,6 +15,7 @@ function determineIntegrationTestStackName(cb) {
   if (branch === 'master') return cb('cumulus-source');
 
   const stacks = {
+    'Charles Huang': 'ch-ci-tf',
     'Jenny Liu': 'jl-rds',
     jennyhliu: 'jl-rds',
     kkelly51: 'kk2',

diff --git a/docs/data-cookbooks/cnm-workflow.md b/docs/data-cookbooks/cnm-workflow.md
@@ -71,7 +71,7 @@ The following are steps that are required to set up your Cumulus instance to run
 
 In this example, we're going to trigger a workflow by creating a Kinesis rule and sending a record to a Kinesis stream.
 
-The following [workflow definition](workflows/README.md) should be added to a new `.tf` workflow resource (e.g. `cnm_workflow.tf`) in your deployment directory.  For the complete CNM workflow example, see [examples/cumulus-tf/kinesis_trigger_test_workflow.tf](https://github.com/nasa/cumulus/blob/master/example/cumulus-tf/kinesis_trigger_test_workflow.tf).
+The following [workflow definition](workflows/README.md) should be added to a new `.tf` workflow resource (e.g. `cnm_workflow.tf`) in your deployment directory.  For the complete CNM workflow example, see [examples/cumulus-tf/cnm_workflow.tf](https://github.com/nasa/cumulus/blob/master/example/cumulus-tf/cnm_workflow.tf).
 
 Add the following to the new terraform file in your deployment directory, updating the following:
 

diff --git a/example/cumulus-tf/cnm_response_task.tf b/example/cumulus-tf/cnm_response_task.tf
@@ -1,5 +1,5 @@
 locals {
-  cnm_response_version  = "2.0.2"
+  cnm_response_version  = "2.0.3"
   cnm_response_filename = "cnmResponse-${local.cnm_response_version}.zip"
 }
 

diff --git a/example/cumulus-tf/cnm_to_cma_task.tf b/example/cumulus-tf/cnm_to_cma_task.tf
@@ -1,5 +1,5 @@
 locals {
-  cnm_to_cma_version  = "1.5.3"
+  cnm_to_cma_version  = "1.5.4"
   cnm_to_cma_filename = "cnmToGranule-${local.cnm_to_cma_version}.zip"
 }
 

diff --git a/...tf/kinesis_trigger_test_workflow.asl.json → example/cumulus-tf/cnm_workflow.asl.json b/...tf/kinesis_trigger_test_workflow.asl.json → example/cumulus-tf/cnm_workflow.asl.json
@@ -1,5 +1,5 @@
 {
-  "Comment": "Tests Workflow from Kinesis Stream",
+  "Comment": "Tests CNM workflow",
   "StartAt": "TranslateMessage",
   "States": {
     "TranslateMessage": {
@@ -380,7 +380,7 @@
             "distribution_endpoint": "{$.meta.distribution_endpoint}",
             "response-endpoint": "{$.meta.cnmResponseStream}",
             "region": "us-east-1",
-            "type": "kinesis",
+            "type": "{$.meta.cnmResponseMethod}",
             "WorkflowException": "{$.exception}",
             "cumulus_message": {
               "outputs": [
@@ -434,7 +434,7 @@
             "OriginalCNM": "{$.meta.cnm}",
             "response-endpoint": "{$.meta.cnmResponseStream}",
             "region": "us-east-1",
-            "type": "kinesis",
+            "type": "{$.meta.cnmResponseMethod}",
             "WorkflowException": "{$.exception}",
             "cumulus_message": {
               "outputs": [

diff --git a/...mulus-tf/kinesis_trigger_test_workflow.tf → example/cumulus-tf/cnm_workflow.tf b/...mulus-tf/kinesis_trigger_test_workflow.tf → example/cumulus-tf/cnm_workflow.tf
@@ -1,4 +1,4 @@
-module "kinesis_trigger_test_workflow" {
+module "cnm_workflow" {
   depends_on = [
     aws_lambda_function.cnm_response_task,
     aws_lambda_function.cnm_to_cma_task
@@ -7,13 +7,13 @@ module "kinesis_trigger_test_workflow" {
   source = "../../tf-modules/workflow"
 
   prefix          = var.prefix
-  name            = "KinesisTriggerTest"
+  name            = "CNMExampleWorkflow"
   workflow_config = module.cumulus.workflow_config
   system_bucket   = var.system_bucket
   tags            = local.tags
 
   state_machine_definition = templatefile(
-    "${path.module}/kinesis_trigger_test_workflow.asl.json",
+    "${path.module}/cnm_workflow.asl.json",
     {
       cnm_to_cma_task_arn: aws_lambda_function.cnm_to_cma_task.arn,
       cnm_response_task_arn: aws_lambda_function.cnm_response_task.arn,

diff --git a/example/cumulus-tf/main.tf b/example/cumulus-tf/main.tf
@@ -162,20 +162,17 @@ module "cumulus" {
   # Archive API settings
   token_secret = var.token_secret
   archive_api_users = [
-    "aortega527",
+    "chuang14",
+    "ds_jennifertran",
     "jasmine",
     "jennyhliu",
     "jmccoy_uat",
-    "kbaynes",
     "kkelly",
     "kovarik",
-    "lfrederick",
-    "matthewsavoie",
-    "mboyd",
     "menno.vandiermen",
     "mobrien84",
-    "npauzenga",
-    "ds_jennifertran"
+    "nnageswa",
+    "npauzenga"
   ]
   archive_api_url             = var.archive_api_url
   archive_api_port            = var.archive_api_port

diff --git a/example/data/collections/ASCATB-L2-Coastal/ASCATB-L2-Coastal.json b/example/data/collections/ASCATB-L2-Coastal/ASCATB-L2-Coastal.json
@@ -0,0 +1,32 @@
+{
+  "duplicateHandling": "replace",
+  "files": [
+    {
+      "bucket": "protected",
+      "regex": "^ascat_[0-9]{8}_[0-9]{6}_metopb_[0-9]{5}_eps_o_coa_[0-9]{4}_ovw\\.l2\\.nc$",
+      "sampleFileName": "ascat_20121029_010301_metopb_00588_eps_o_coa_2101_ovw.l2.nc",
+      "type": "data"
+    },
+    {
+      "bucket": "private",
+      "regex": "^ascat_[0-9]{8}_[0-9]{6}_metopb_[0-9]{5}_eps_o_coa_[0-9]{4}_ovw\\.l2.*\\.cmr\\.json$",
+      "sampleFileName": "ascat_20121029_010301_metopb_00588_eps_o_coa_2101_ovw.l2.cmr.json",
+      "type": "metadata"
+    },
+    {
+      "bucket": "protected",
+      "regex": "^ascat_[0-9]{8}_[0-9]{6}_metopb_[0-9]{5}_eps_o_coa_[0-9]{4}_ovw\\.l2\\.nc\\.dmrpp$",
+      "sampleFileName": "ascat_20121029_010301_metopb_00588_eps_o_coa_2101_ovw.l2.nc.dmrpp",
+      "type": "metadata"
+    }
+  ],
+  "granuleId": "^ascat_[0-9]{8}_[0-9]{6}_metopb_[0-9]{5}_eps_o_coa_[0-9]{4}_ovw\\.l2$",
+  "granuleIdExtraction": "^(.*)(\\.nc|\\.cmr)",
+  "meta": {
+    "granuleMetadataFileExtension": ".cmr.json"
+  },
+  "name": "ASCATB-L2-Coastal",
+  "sampleFileName": "ascat_20121029_010301_metopb_00588_eps_o_coa_2101_ovw.l2.nc",
+  "url_path": "{cmrMetadata.CollectionReference.ShortName}",
+  "version": "Operational/Near-Real-Time"
+}
diff --git a/example/deployments/cumulus/ch-ci-tf.tfvars b/example/deployments/cumulus/ch-ci-tf.tfvars
@@ -0,0 +1,2 @@
+prefix = "ch-ci-tf"
+key_name = "chuang14"
diff --git a/example/deployments/data-persistence/ch-ci-tf.tfvars b/example/deployments/data-persistence/ch-ci-tf.tfvars
@@ -0,0 +1 @@
+prefix = "ch-ci-tf"
diff --git a/example/deployments/db-migration/ch-ci-tf.tfvars b/example/deployments/db-migration/ch-ci-tf.tfvars
@@ -0,0 +1 @@
+prefix = "ch-ci-tf"
diff --git a/example/lambdas/asyncOperations/package.json b/example/lambdas/asyncOperations/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-async-operations",
-  "version": "12.0.2",
+  "version": "12.0.3",
   "description": "AsyncOperations Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/ftpPopulateTestLambda/package.json b/example/lambdas/ftpPopulateTestLambda/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/ftp-populate-test-lambda",
-  "version": "12.0.2",
+  "version": "12.0.3",
   "description": "FTP Population Utility Lambda",
   "main": "index.js",
   "private": true,
@@ -19,12 +19,12 @@
     "access": "private"
   },
   "dependencies": {
-    "@cumulus/api": "12.0.2",
-    "@cumulus/api-client": "12.0.2",
-    "@cumulus/common": "12.0.2",
-    "@cumulus/integration-tests": "12.0.2",
-    "@cumulus/logger": "12.0.2",
-    "@cumulus/test-data": "12.0.2",
+    "@cumulus/api": "12.0.3",
+    "@cumulus/api-client": "12.0.3",
+    "@cumulus/common": "12.0.3",
+    "@cumulus/integration-tests": "12.0.3",
+    "@cumulus/logger": "12.0.3",
+    "@cumulus/test-data": "12.0.3",
     "aws-sdk": "^2.585.0",
     "fs-extra": "^9.0.0",
     "js-yaml": "^3.13.1",

diff --git a/example/lambdas/python-processing/package.json b/example/lambdas/python-processing/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-process-activity",
   "private": true,
-  "version": "12.0.2",
+  "version": "12.0.3",
   "description": "Python reference activity",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-activity",
   "repository": {

diff --git a/example/lambdas/python-reference-activity/package.json b/example/lambdas/python-reference-activity/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-reference-activity",
   "private": true,
-  "version": "12.0.2",
+  "version": "12.0.3",
   "description": "Python reference activity",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-activity",
   "repository": {

diff --git a/example/lambdas/python-reference-task/package.json b/example/lambdas/python-reference-task/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-reference-task",
   "private": true,
-  "version": "12.0.2",
+  "version": "12.0.3",
   "description": "Python reference task",
   "main": "index.js",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-task",

diff --git a/example/lambdas/s3AccessTest/package.json b/example/lambdas/s3AccessTest/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-s3-access",
-  "version": "12.0.2",
+  "version": "12.0.3",
   "description": "S3 Access Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/snsS3Test/package.json b/example/lambdas/snsS3Test/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-sns-s3",
-  "version": "12.0.2",
+  "version": "12.0.3",
   "description": "SNS to S3 Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/versionUpTest/package.json b/example/lambdas/versionUpTest/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-version-up",
-  "version": "12.0.2",
+  "version": "12.0.3",
   "description": "Version Up Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/package.json b/example/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/cumulus-integration-tests",
-  "version": "12.0.2",
+  "version": "12.0.3",
   "description": "Cumulus Integration Test Deployment",
   "private": true,
   "main": "index.js",
@@ -45,32 +45,32 @@
     ]
   },
   "dependencies": {
-    "@cumulus/api": "12.0.2",
-    "@cumulus/api-client": "12.0.2",
-    "@cumulus/async-operations": "12.0.2",
-    "@cumulus/aws-client": "12.0.2",
-    "@cumulus/checksum": "12.0.2",
-    "@cumulus/cmr-client": "12.0.2",
-    "@cumulus/cmrjs": "12.0.2",
-    "@cumulus/common": "12.0.2",
-    "@cumulus/discover-granules": "12.0.2",
-    "@cumulus/discover-pdrs": "12.0.2",
-    "@cumulus/files-to-granules": "12.0.2",
-    "@cumulus/hello-world": "12.0.2",
-    "@cumulus/ingest": "12.0.2",
-    "@cumulus/integration-tests": "12.0.2",
-    "@cumulus/message": "12.0.2",
-    "@cumulus/move-granules": "12.0.2",
-    "@cumulus/parse-pdr": "12.0.2",
-    "@cumulus/pdr-status-check": "12.0.2",
-    "@cumulus/post-to-cmr": "12.0.2",
-    "@cumulus/queue-granules": "12.0.2",
-    "@cumulus/queue-pdrs": "12.0.2",
-    "@cumulus/sf-sqs-report": "12.0.2",
-    "@cumulus/sync-granule": "12.0.2",
-    "@cumulus/test-processing": "12.0.2"
+    "@cumulus/api": "12.0.3",
+    "@cumulus/api-client": "12.0.3",
+    "@cumulus/async-operations": "12.0.3",
+    "@cumulus/aws-client": "12.0.3",
+    "@cumulus/checksum": "12.0.3",
+    "@cumulus/cmr-client": "12.0.3",
+    "@cumulus/cmrjs": "12.0.3",
+    "@cumulus/common": "12.0.3",
+    "@cumulus/discover-granules": "12.0.3",
+    "@cumulus/discover-pdrs": "12.0.3",
+    "@cumulus/files-to-granules": "12.0.3",
+    "@cumulus/hello-world": "12.0.3",
+    "@cumulus/ingest": "12.0.3",
+    "@cumulus/integration-tests": "12.0.3",
+    "@cumulus/message": "12.0.3",
+    "@cumulus/move-granules": "12.0.3",
+    "@cumulus/parse-pdr": "12.0.3",
+    "@cumulus/pdr-status-check": "12.0.3",
+    "@cumulus/post-to-cmr": "12.0.3",
+    "@cumulus/queue-granules": "12.0.3",
+    "@cumulus/queue-pdrs": "12.0.3",
+    "@cumulus/sf-sqs-report": "12.0.3",
+    "@cumulus/sync-granule": "12.0.3",
+    "@cumulus/test-processing": "12.0.3"
   },
   "devDependencies": {
-    "@cumulus/test-data": "12.0.2"
+    "@cumulus/test-data": "12.0.3"
   }
 }
diff --git a/example/scripts/generate_ingest/package.json b/example/scripts/generate_ingest/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/generate_ingest",
   "private": true,
-  "version": "12.0.2",
+  "version": "12.0.3",
   "description": "Script to generate test data for scaled ingest",
   "keywords": [
     "GIBS",
@@ -22,8 +22,8 @@
     "directory": "packages/types"
   },
   "dependencies": {
-    "@cumulus/aws-client": "12.0.2",
-    "@cumulus/common": "12.0.2"
+    "@cumulus/aws-client": "12.0.3",
+    "@cumulus/common": "12.0.3"
   },
   "author": "Cumulus Authors",
   "license": "Apache-2.0"

diff --git a/example/scripts/lib/package.json b/example/scripts/lib/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/example-lib",
   "private": true,
-  "version": "12.0.2",
+  "version": "12.0.3",
   "description": "example project libs",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/scripts/lib",
   "engines": {