v18.2.0-alpha.0

Release v18.2.0-alpha.0

v16.1.2

Release v16.1.2

Please note changes in 16.1.2 may not yet be released in future versions, as this
is a backport/patch release on the 16.x series of releases. Updates that are
included in the future will have a corresponding CHANGELOG entry in future releases.

Dashboard Requirement

Cumulus Dashboard v12.0.1 https://github.com/nasa/cumulus-dashboard/releases/tag/v12.0.1

Added

• CUMULUS-3218
  • Added optional maxDownloadTime field to provider schema
  • Added max_download_time column to PostgreSQL providers table
  • Updated @cumulus/ingest/lock to check expired locks based on provider.maxDownloadTime

Fixed

• @AWS-SDK upgrade
  • Fixed TS compilation error on aws-client package caused by @aws-sdk/client-dynamodb 3.433.0 upgrade
  • Updated mapping for collection Elasticsearch records to prevent dynamic field for keys under meta.
• CUMULUS-3286
  • Fixed @cumulus/cmrjs/cmr-utils/getGranuleTemporalInfo and @cumulus/message/Granules/getGranuleCmrTemporalInfo
    to handle non-existing cmr file.
  • Updated mapping for granule and deletedgranule Elasticsearch records to prevent dynamic field for keys under
    queryFields.
• CUMULUS-3293
  • Process Dead Letter Archive is fixed to properly copy objects from /sqs/ to /failed-sqs/ location
• CUMULUS-3393
  • Fixed PUT collection endpoint to update collection configuration in S3.
• CUMULUS-3467
  • Added childWorkflowMeta to QueueWorkflow task configuration

v18.1.0

Dashboard Requirement

Note that this release contains breaking changes for the Cumulus Dashboard. You must use the Cumulus Dashboard v12.1.0 or greater

MIGRATION notes

Rules API Endpoint Versioning

As part of the work on CUMULUS-3095, we have added a required header for the
rules PUT/PATCH endpoints -- to ensure that older clients/utilities do not
unexpectedly make destructive use of those endpoints, a validation check of a
header value against supported versions has been implemented.

Moving forward, if a breaking change is made to an existing endpoint that
requires user updates, as part of that update we will set the current version of
the core API and require a header that confirms the client is compatible with
the version required or greater.

In this instance, the rules PUT/PATCH
endpoints will require a Cumulus-API-Version value of at least 2.

 curl --request PUT https://example.com/rules/repeat_test\
 --header 'Cumulus-API-Version: 2'\
 --header 'Content-Type: application/json'\
 --header 'Authorization: Bearer ReplaceWithToken'\
 --data ...

Users/clients that do not make use of these endpoints will not be impacted.

Breaking Changes

• CUMULUS-3427
  • Changed the naming conventions for memory size and timeouts configuration to simply the lambda name

Notable Changes

• CUMULUS-3095
  • Added PATCH rules endpoint to update rule which works as the existing PUT endpoint.
  • Updated PUT rules endpoint to replace rule.

Added

• CUMULUS-3218
  • Added optional maxDownloadTime field to provider schema
  • Added max_download_time column to PostgreSQL providers table
  • Updated @cumulus/ingest/lock to check expired locks based on provider.maxDownloadTime

Changed

• CUMULUS-3095
  • Updated @cumulus/api-client/rules to havereplaceRule and updateRule methods.
  • Updated mapping for rule Elasticsearch records to prevent dynamic field for keys under
    meta and payload, and fixed rule field mapping.
• CUMULUS-3351
  • Updated constructOnlineAccessUrls() to group CMR online access URLs by link type.
• CUMULUS-3377
  • Added configuration option to cumulus-tf/terraform.tfvars to include sns:Subscribe access policy for
    executions, granules, collections, and PDRs report topics.
• CUMULUS-3392
  • Modify cloudwatch rule by deleting custom
• CUMULUS-3434
  • Updated @cumulus/orca-recovery-adapter task to output both input granules and recovery output.
  • Updated example/cumulus-tf/orca.tf to use v9.0.0.

Fixed

• CUMULUS-3095
  • Added back rule schema validation which is missing after RDS phase 3.
  • Fixed a bug for creating rule with tags.
• CUMULUS-3286
  • Fixed @cumulus/cmrjs/cmr-utils/getGranuleTemporalInfo and @cumulus/message/Granules/getGranuleCmrTemporalInfo
    to handle non-existing cmr file.
  • Updated mapping for granule and deletedgranule Elasticsearch records to prevent dynamic field for keys under
    queryFields.
  • Updated mapping for collection Elasticsearch records to prevent dynamic field for keys under meta.
• CUMULUS-3393
  • Fixed PUT collection endpoint to update collection configuration in S3.
• CUMULUS-3427
  • Fixed issue where some lambda and task memory sizes and timeouts were not configurable
• @AWS-SDK upgrade
  • Fixed TS compilation error on aws-client package caused by @aws-sdk/client-dynamodb 3.433.0 upgrade

v18.0.0

Release v18.0.0

Notable Changes

• CUMULUS-3259

  • Updated Terraform version from 0.13.6 to 1.5.3. Please see the instructions to upgrade your deployments.

• CUMULUS-3270

  • update python lambdas to use python3.10
  • update dependencies to use python3.10 including cumulus-message-adapter, cumulus-message-adapter-python and cumulus-process-py

Changes

• CUMULUS-3366
  • Added logging to the collectionRuleMatcher Rules Helper, which is used by the sqs-message-consumer and message-consumer Lambdas,
    to report when an incoming message's collection does not match any rules.

v17.0.0

Release 17.0.0

Notable Changes

• CUMULUS-3258
  • @cumulus/api is now compatible only with Orca >= 8.1.0. Prior versions of
    Orca are not compatible with Cumulus 17+
  • Updated all hashicorp terraform AWS provider configs to ~> 5.0
    • Upstream/downstream terraform modules will need to utilize an AWS provider
      that matches this range

MIGRATION notes

• This release updates the hashicorp/aws provider required by Cumulus to ~> 5.0
  which in turn requires updates to all modules deployed with Core in the same stack
  to use a compatible provider version.
• This update is not compatible with prior stack states - Terraform will not
  allow redeployment of a prior version of Cumulus using an older version of
  the provider. Please be sure to validate the install changeset is what you
  expect prior to upgrading to this version.
• Upgrading Cumulus to v17 from prior versions should only require the usual
  terraform init/apply steps. As always be sure to inspect the terraform plan or
  terraform apply changeset to ensure the changes between providers are what
  you're expecting for all modules you've chosen to deploy with Cumulus

Breaking Changes

• CUMULUS-3258

  • Update @cumulus/api/lib/orca/getOrcaRecoveryStatusByGranuleCollection
    to @cumulus/api/lib/orca/getOrcaRecoveryStatusByGranuleIdAndCollection and
    add collectionId to arguments to support Orca v8+ required use of
    collectionId

  • Updated all terraform AWS providers to ~> 5.0

Changes

• CUMULUS-3258
  • Update all Core integration tests/integrations to be compatible with Orca >=
    v8.1.0 only

Fixed

• CUMULUS-3319
  • Removed @cumulus/api/models/schema and changed all references to
    @cumulus/api/lib/schema in docs and related models
  • Removed @cumulus/api/models/errors.js
  • Updated API granule write logic to cause postgres schema/db write failures on an individual granule file write to result in a thrown error/400 return instead of a 200 return and a 'silent' update of the granule to failed status.
  • Update api/lib/_writeGranule/_writeGranulefiles logic to allow for schema failures on individual granule writes via an optional method parameter in _writeGranules, and an update to the API granule write calls.
  • Updated thrown error to include information related to automatic failure behavior in addition to the stack trace.

v16.1.1

Release 16.1.1

Compatible Orca Version

• v6.0.3 or v6.04. The orca related ticket work is for future orca releases.

Notable Changes

• The async_operation_image property of cumulus module should be updated to pull
  the ECR image for cumuluss/async-operation:47

Added

• CUMULUS-3298
  • Added extra time to the buffer for replacing the launchpad token before it expires to alleviate CMR error messages
• CUMULUS-3220
  • Created a new send-pan task
• CUMULUS-3287
  • Added variable to allow the aws_ecs_task_definition health check to be configurable.
  • Added clarity to how the bucket field needs to be configured for the move-granules task definition

Changed

• Security upgrade node from 14.19.3-buster to 14.21.1-buster
• CUMULUS-2985
  • Changed onetime rules RuleTrigger to only execute when the state is ENABLED and updated documentation to reflect the change
  • Changed the invokeRerun function to only re-run enabled rules
• CUMULUS-3188
  • Updated QueueGranules to support queueing granules that meet the required API granule schema.
  • Added optional additional properties to queue-granules input schema
• CUMULUS-3252
  • Updated example/cumulus-tf/orca.tf to use orca v8.0.1
  • Added cumulus task @cumulus/orca-copy-to-archive-adapter, and add the task to tf-modules/ingest
  • Updated tf-modules/cumulus module to take variable orca_lambda_copy_to_archive_arn and pass to tf-modules/ingest
  • Updated example/cumulus-tf/ingest_and_publish_granule_with_orca_workflow.tf CopyToGlacier (renamed to CopyToArchive) step to call
    orca_copy_to_archive_adapter_task
• CUMULUS-3253
  • Added cumulus task @cumulus/orca-recovery-adapter, and add the task to tf-modules/ingest
  • Updated tf-modules/cumulus module to take variable orca_sfn_recovery_workflow_arn and pass to tf-modules/ingest
  • Added example/cumulus-tf/orca_recovery_adapter_workflow.tf, OrcaRecoveryAdapterWorkflow workflow has OrcaRecoveryAdapter task
    to call the ORCA recovery step-function.
  • Updated example/data/collections/ collection configuration meta.granuleRecoveryWorkflow to use OrcaRecoveryAdapterWorkflow
• CUMULUS-3215
  • Create reconciliation reports will properly throw errors and set the async
    operation status correctly to failed if there is an error.
  • Knex calls relating to reconciliation reports will retry if there is a
    connection terminated unexpectedly error
  • Improved logging for async operation
  • Set default async_operation_image_version to 47
• CUMULUS-3024
  • Combined unit testing of @cumulus/api/lib/rulesHelpers to a single test file
    api/tests/lib/test-rulesHelpers and removed extraneous test files.
• CUMULUS-3209
  • Apply brand color with high contrast settings for both (light and dark) themes.
  • Cumulus logo can be seen when scrolling down.
  • "Back to Top" button matches the brand color for both themes.
  • Update "note", "info", "tip", "caution", and "warning" components to new admonition styling.
  • Add updated arch diagram for both themes.
• CUMULUS-3203
  • Removed ACL setting of private on S3.multipartCopyObject() call
  • Removed ACL setting of private for s3PutObject()
  • Removed ACL confguration on sync-granules task
  • Update documentation on dashboard deployment to exclude ACL public-read setting
• CUMULUS-3245
  • Update SQS consumer logic to catch ExecutionAlreadyExists error and
    delete SQS message accordingly.
  • Add ReportBatchItemFailures to event source mapping start_sf_mapping
• Added missing name to throttle_queue_watcher Cloudwatch event in throttled-queue.tf

Fixed

• CUMULUS-2625
  • Optimized heap memory and api load in queue-granules task to scale to larger workloads.
• CUMULUS-3265
  • Fixed @cumulus/api getGranulesForPayload function to query cloud metrics es when needed.

v16.0.0

Release v16.0.0

Dashboard Requirement

Note that this release contains breaking changes for the Cumulus Dashboard. You must use the Cumulus Dashboard v12.0.1 https://github.com/nasa/cumulus-dashboard/releases/tag/v12.0.1

Notable Changes

• The async_operation_image property of cumulus module should be updated to pull
  the ECR image for cumuluss/async-operation:46

MIGRATION notes

PI release version

When updating directly to v16 from prior releases older that V15, please make sure to
read through all prior release notes.

Notable migration concerns since the last PI release version (11.1.x):

• [v14.1.0] - Postgres compatibility update to Aurora PostgreSQL 11.13.
• [v13.1.0] - Postgres update to add files_granule_cumulus_id_index to the
  files table may require manual steps depending on load.

RDS Phase 3 migration notes

This release includes updates that remove existing DynamoDB tables as part of
release deployment process. This release cannot be properly rolled back in
production as redeploying a prior version of Cumulus will not recover the
associated Dynamo tables.

Please read the full change log for RDS Phase 3 and consult the RDS Phase 3 update
documentation

API Endpoint Versioning

As part of the work on CUMULUS-3072, we have added a required header for the
granule PUT/PATCH endpoints -- to ensure that older clients/utilities do not
unexpectedly make destructive use of those endpoints, a validation check of a
header value against supported versions has been implemented.

Moving forward, if a breaking change is made to an existing endpoint that
requires user updates, as part of that update we will set the current version of
the core API and require a header that confirms the client is compatible with
the version required or greater.

In this instance, the granule PUT/PATCH
endpoints will require a Cumulus-API-Version value of at least 2.

 curl --request PUT https://example.com/granules/granuleId.A19990103.006.1000\
 --header 'Cumulus-API-Version: 2'\
 --header 'Authorization: Bearer ReplaceWithToken'\
 --data ...

Users/clients that do not make use of these endpoints will not be impacted.

RDS Phase 3

Breaking Changes

• CUMULUS-2688
  • Updated bulk operation logic to use collectionId in addition to granuleId to fetch granules.
  • Tasks using the bulk-operation Lambda should provide collectionId and granuleId e.g. { granuleId: xxx, collectionId: xxx }
• CUMULUS-2856
  • Update execution PUT endpoint to no longer respect message write constraints and update all values passed in

Changed

• CUMULUS-3282
  • Updated internal granule endpoint parameters from :granuleName to :granuleId
    for maintenance/consistency reasons
• CUMULUS-2312 - RDS Migration Epic Phase 3
  • CUMULUS-2645
    • Removed unused index functionality for all tables other than
      ReconciliationReportsTable from dbIndexer lambda
  • CUMULUS-2398
    • Remove all dynamoDB updates for @cumulus/api/ecs/async-operation/*
    • Updates all api endpoints with updated signature for
      asyncOperationsStart calls
    • Remove all dynamoDB models calls from async-operations api endpoints
  • CUMULUS-2801
    • Move getFilesExistingAtLocationfrom api granules model to api/lib, update granules put
      endpoint to remove model references
  • CUMULUS-2804
    • Updates api/lib/granule-delete.deleteGranuleAndFiles:
      • Updates dynamoGranule -> apiGranule in the signature and throughout the dependent code
      • Updates logic to make apiGranule optional, but pgGranule required, and
        all lookups use postgres instead of ES/implied apiGranule values
      • Updates logic to make pgGranule optional - in this case the logic removes the entry from ES only
    • Removes all dynamo model logic from api/endpoints/granules
    • Removes dynamo write logic from api/lib/writeRecords.*
    • Removes dynamo write logic from api/lib/ingest.*
    • Removes all granule model calls from api/lambdas/bulk-operations and any dependencies
    • Removes dynamo model calls from api/lib/granule-remove-from-cmr.unpublishGranule
    • Removes Post Deployment execution check from sf-event-sqs-to-db-records
    • Moves describeGranuleExecution from api granule model to api/lib/executions.js
  • CUMULUS-2806
    • Remove DynamoDB logic from executions POST endpoint
    • Remove DynamoDB logic from sf-event-sqs-to-db-records lambda execution writes.
    • Remove DynamoDB logic from executions PUT endpoint
  • CUMULUS-2808
    • Remove DynamoDB logic from executions DELETE endpoint
  • CUMULUS-2809
    • Remove DynamoDB logic from providers PUT endpoint
    • Updates DB models asyncOperation, provider and rule to return all fields on upsert.
  • CUMULUS-2810
    • Removes addition of DynamoDB record from API endpoint POST /provider/
  • CUMULUS-2811
    • Removes deletion of DynamoDB record from API endpoint DELETE /provider/
  • CUMULUS-2817
    • Removes deletion of DynamoDB record from API endpoint DELETE /collection//
  • CUMULUS-2814
    • Move event resources deletion logic from rulesModel to rulesHelper
  • CUMULUS-2815
    • Move File Config and Core Config validation logic for Postgres Collections from api/models/collections.js to api/lib/utils.js
  • CUMULUS-2813
    • Removes creation and deletion of DynamoDB record from API endpoint POST /rules/
  • CUMULUS-2816
    • Removes addition of DynamoDB record from API endpoint POST /collections
  • CUMULUS-2797
    • Move rule helper functions to separate rulesHelpers file
  • CUMULUS-2821
    • Remove DynamoDB logic from sfEventSqsToDbRecords lambda
  • CUMULUS-2856
    • Update API/Message write logic to handle nulls as deletion in execution PUT/message write logic

Added

• CUMULUS-2312 - RDS Migration Epic Phase 3
  • CUMULUS-2813
    • Added function create in the db model for Rules
      to return an array of objects containing all columns of the created record.
  • CUMULUS-2812
    • Move event resources logic from rulesModel to rulesHelper
  • CUMULUS-2820
    • Remove deletion of DynamoDB record from API endpoint DELETE /pdr/
  • CUMULUS-2688
    • Add new endpoint to fetch granules by collectionId as well as granuleId: GET /collectionId/granuleId
    • Add new endpoints to update and delete granules by collectionId as well as
      granuleId

Removed

• CUMULUS-2994
  • Delete code/lambdas that publish DynamoDB stream events to SNS
• CUMULUS-3226
  • Removed Dynamo Async Operations table
• CUMULUS-3199
  • Removed DbIndexer lambda and all associated terraform resources
• CUMULUS-3009
  • Removed Dynamo PDRs table
• CUMULUS-3008
  • Removed DynamoDB Collections table
• CUMULUS-2815
  • Remove update of DynamoDB record from API endpoint PUT /collections//
• CUMULUS-2814
  • Remove DynamoDB logic from rules DELETE endpoint
• CUMULUS-2812
  • Remove DynamoDB logic from rules PUT endpoint
• CUMULUS-2798
  • Removed AsyncOperations model
• CUMULUS-2797
• CUMULUS-2795
  • Removed API executions model
• CUMULUS-2796
  • Remove API pdrs model and all related test code
  • Remove API Rules model and all related test code
• CUMULUS-2794
  • Remove API Collections model and all related test code
  • Remove lambdas/postgres-migration-count-tool, api/endpoints/migrationCounts and api-client/migrationCounts
  • Remove lambdas/data-migration1 tool
  • Remove lambdas/data-migration2 and
    lambdas/postgres-migration-async-operation
• CUMULUS-2793
  • Removed Provider Dynamo model and related test code
• CUMULUS-2792
  • Remove API Granule model and all related test code
  • Remove granule-csv endpoint
• CUMULUS-2645
  • Removed dynamo structural migrations and related code from @cumulus/api
  • Removed executeMigrations lambda
  • Removed granuleFilesCacheUpdater lambda
  • Removed dynamo files table from data-persistence module. This table and
    all of its data will be removed on deployment.

Added

• CUMULUS-3072
  • Added replaceGranule to @cumulus/api-client/granules to add usage of the
    updated RESTful PUT logic
• CUMULUS-3121
  • Added a map of variables for the cloud_watch_log retention_in_days for the various cloudwatch_log_groups, as opposed to keeping them hardcoded at 30 days. Can be configured by adding the _<cloudwatch_log_group_name>_log_retention value in days to the cloudwatch_log_retention_groups map variable
• CUMULUS-3201
  • Added support for sha512 as checksumType for LZARDs backup task.

Changed

• CUMULUS-3315
  • Updated @cumulus/api-client/granules.bulkOperation to remove ids
    parameter in favor of granules parameter, in the form of a
    @cumulus/types/ApiGranule that requires the following keys: [granuleId, collectionId]
• CUMULUS-3307
  • Pinned cumulus dependency on pg to v8.10.x
• CUMULUS-3279
  • Updated core dependencies on xml2js to v0.5.0
  • Forcibly updated downstream dependency for xml2js in saml2-js to
    v0.5.0
  • Added audit-ci CVE override until July 1 to allow for Core package releases
• CUMULUS-3106
  • Updated localstack version to 1.4.0 and removed 'skip' from all skipped tests
• CUMULUS-3115
  • Fixed DiscoverGranules' workflow's duplicateHandling when set to skip or error to stop retrying
    after receiving a 404 Not Found Response Error from the cumulus-api.
• CUMULUS-3165
  • Update example/cumulus-tf/orca.tf to use orca v6...

v16.1.0-alpha.0

Targeted alpha release from jk/terraform-provider-with-rel-16 for partner use. NON-PRODUCTION RELEASE.

v15.0.4

Changed

• CUMULUS-3307
  • Pinned cumulus dependency on pg to v8.10.x

Fixed

• CUMULUS-3115
  • Fixed DiscoverGranules' workflow's duplicateHandling when set to skip or error to stop retrying
    after receiving a 404 Not Found Response Error from the cumulus-api.
• CUMULUS-3315
  • Update CI scripts to use shell logic/GNU timeout to bound test timeouts
    instead of NPM parallel package, as timeouts were not resulting in
    integration test failure
• CUMULUS-3223
  • Update @cumulus/cmrjs/cmr-utils.getGranuleTemporalInfo to handle the error when the cmr file s3url is not available
  • Update sfEventSqsToDbRecords lambda to return partial batch failure,
    and only reprocess messages when cumulus message can't be retrieved from the execution events.
  • Update @cumulus/cumulus-message-adapter-js to 2.0.5 for all cumulus tasks

v15.0.3

Fixed

• CUMULUS-3243
  • Updated granule delete logic to delete granule which is not in DynamoDB
  • Updated granule unpublish logic to handle granule which is not in DynamoDB and/or CMR