v9.5.0

BREAKING CHANGES

• Removed logs record type from mappings from Elasticsearch. This change should not have
  any adverse impact on existing deployments, even those which still contain logs records,
  but technically it is a breaking change to the Elasticsearch mappings.
• Changed @cumulus/api-client/asyncOperations.getAsyncOperation to return parsed JSON body
  of response and not the raw API endpoint response

Added

• CUMULUS-2670

  • Updated core cumulus module to take lambda_timeouts string map variable that allows timeouts of ingest tasks to be configurable. Allowed properties for the mapping include:
  • discover_granules_task_timeout
  • discover_pdrs_task_timeout
  • hyrax_metadata_update_tasks_timeout
  • lzards_backup_task_timeout
  • move_granules_task_timeout
  • parse_pdr_task_timeout
  • pdr_status_check_task_timeout
  • post_to_cmr_task_timeout
  • queue_granules_task_timeout
  • queue_pdrs_task_timeout
  • queue_workflow_task_timeout
  • sync_granule_task_timeout

• CUMULUS-2575

  • Adds POST /granules API endpoint to create a granule
  • Adds helper createGranule to @cumulus/api-client

• CUMULUS-2577

  • Adds POST /executions endpoint to create an execution

• CUMULUS-2578

  • Adds PUT /executions endpoint to update an execution

• CUMULUS-2592

  • Adds logging when messages fail to be added to queue

• CUMULUS-2644

  • Pulled delete method for granules-executions.ts implemented as part of CUMULUS-2306
    from the RDS-Phase-2 feature branch in support of CUMULUS-2644.
  • Pulled erasePostgresTables method in serve.js implemented as part of CUMULUS-2644,
    and CUMULUS-2306 from the RDS-Phase-2 feature branch in support of CUMULUS-2644
  • Added resetPostgresDb method to support resetting between integration test suite runs

Changed

• Updated processDeadLetterArchive Lambda to return an object where
  processingSucceededKeys is an array of the S3 keys for successfully
  processed objects and processingFailedKeys is an array of S3 keys
  for objects that could not be processed

• Updated async operations to handle writing records to the databases
  when output of the operation is undefined

• CUMULUS-2644

  • Moved migration directory from the db-migration-lambda to the db package and
    updated unit test references to migrationDir to be pulled from @cumulus/db
  • Updated @cumulus/api/bin/serveUtils to write records to PostgreSQL tables

• CUMULUS-2575

  • Updates model/granule to allow a granule created from API to not require an
    execution to be associated with it. This is a backwards compatible change
    that will not affect granules created in the normal way.
  • Updates @cumulus/db/src/model/granules functions get and exists to
    enforce parameter checking so that requests include either (granule_id
    and collection_cumulus_id) or (cumulus_id) to prevent incorrect results.
  • @cumulus/message/src/Collections.deconstructCollectionId has been
    modified to throw a descriptive error if the input collectionId is
    undefined rather than TypeError: Cannot read property 'split' of undefined. This function has also been updated to throw descriptive errors
    if an incorrectly formated collectionId is input.

v9.4.0

Notable changes

• @cumulus/sync-granule task should now properly handle
  syncing files from HTTP/HTTPS providers where basic auth is
  required and involves a redirect to a different host (e.g.
  downloading files protected by Earthdata Login)

Added

• CUMULUS-2591
  • Adds failedExecutionStepName to failed execution's jsonb error records.
    This is the name of the Step Function step for the last failed event in the
    execution's event history.
• CUMULUS-2548
  • Added allowed_redirects field to PostgreSQL providers table
  • Added allowedRedirects field to DynamoDB <prefix>-providers table
  • Added @cumulus/aws-client/S3.streamS3Upload to handle uploading the contents
    of a readable stream to S3 and returning a promise
• CUMULUS-2373
  • Added replaySqsMessages lambda to replay archived incoming SQS
    messages from S3.
  • Added /replays/sqs endpoint to trigger an async operation for
    the replaySqsMessages lambda.
  • Added unit tests and integration tests for new endpoint and lambda.
  • Added getS3PrefixForArchivedMessage to ingest/sqs package to get prefix
    for an archived message.
  • Added new async_operation type SQS Replay.
• CUMULUS-2460
  • Adds POST /executions/workflows-by-granules for retrieving workflow names common to a set of granules
  • Adds workflowsByGranules to @cumulus/api-client/executions
• CUMULUS-2635
  • Added helper functions:
    • @cumulus/db/translate/file/translateApiPdrToPostgresPdr

Fixed

• CUMULUS-2548
  • Fixed @cumulus/ingest/HttpProviderClient.sync to
    properly handle basic auth when redirecting to a different
    host and/or host with a different port
• CUMULUS-2626
  • Update PDR migration to correctly find Executions by a Dynamo PDR's execution field
• CUMULUS-2635
  • Update data-migration2 to migrate PDRs before migrating granules.
  • Update data-migration2 unit tests testing granules migration to reference
    PDR records to better model the DB schema.
  • Update migratePdrRecord to use translateApiPdrToPostgresPdr function.

Changed

• CUMULUS-2373
  • Updated getS3KeyForArchivedMessage in ingest/sqs to store SQS messages
    by queueName.
• CUMULUS-2630
  • Updates the example/cumulus-tf deployment to change
    archive_api_reserved_concurrency from 2 to 8 to prevent throttling with
    the dashboard.

v9.2.2

Release v9.2.2

Please note changes in 9.2.2 may not yet be released in future versions, as
this is a backport and patch release on the 9.2.x series of releases. Updates that
are included in the future will have a corresponding CHANGELOG entry in future
releases.

Added

• CUMULUS-2635
  • Added helper functions:
    • @cumulus/db/translate/file/translateApiPdrToPostgresPdr

Fixed

• CUMULUS-2635
  • Update data-migration2 to migrate PDRs before migrating granules.
  • Update data-migration2 unit tests testing granules migration to reference
    PDR records to better model the DB schema.
  • Update migratePdrRecord to use translateApiPdrToPostgresPdr function.

v8.1.2

Please note changes in 8.1.2 may not yet be released in future versions, as this
is a backport/patch release on the 8.x series of releases. Updates that are
included in the future will have a corresponding CHANGELOG entry in future releases.

Notable changes

• cmr_custom_host variable for cumulus module can now be used to configure Cumulus to
  integrate with a custom CMR host name and protocol (e.g. http://custom-cmr-host.com). Note
  that you must include a protocol (http:// or https://) if specifying a value for this
  variable.
• @cumulus/sync-granule task should now properly handle
  syncing files from HTTP/HTTPS providers where basic auth is
  required and involves a redirect to a different host (e.g.
  downloading files protected by Earthdata Login)

Added

• CUMULUS-2548
  • Added allowed_redirects field to PostgreSQL providers table
  • Added allowedRedirects field to DynamoDB <prefix>-providers table
  • Added @cumulus/aws-client/S3.streamS3Upload to handle uploading the contents
    of a readable stream to S3 and returning a promise

Changed

• Updated cmr_custom_host variable to accept a full protocol and host name
  (e.g. http://cmr-custom-host.com), whereas it previously only accepted a host name

Fixed

• Fixed bug where cmr_custom_host variable was not properly forwarded into archive, ingest, and sqs-message-remover modules from cumulus module
• CUMULUS-2548
  • Fixed @cumulus/ingest/HttpProviderClient.sync to
    properly handle basic auth when redirecting to a different
    host and/or host with a different port

v9.2.1

Fixed

• CUMULUS-2626
  • Update PDR migration to correctly find Executions by a Dynamo PDR's execution field.

v9.3.0

Breaking Changes

• All API requests made by @cumulus/api-client will now throw an error if the status code
  does not match the expected response (200 for most requests and 202 for a few requests that
  trigger async operations). Previously the helpers in this package would return the response
  regardless of the status code, so you may need to update any code using helpers from this
  package to catch or to otherwise handle errors that you may encounter.
• The Cumulus API Lambda function has now been configured with reserved concurrency to ensure
  availability in a high-concurrency environment. However, this also caps max concurrency which
  may result in throttling errors if trying to reach the Cumulus API multiple times in a short
  period. Reserved concurrency can be configured with the archive_api_reserved_concurrency
  terraform variable on the Cumulus module and increased if you are seeing throttling erorrs.
  The default reserved concurrency value is 8.

Notable Changes

• The Cumulus Distribution API can be configured to use the AWS Cognito OAuth client. This API
  can be used instead of the Thin Egress App. See the Migrate from TEA deployment to Cumulus Distribution
• cmr_custom_host variable for cumulus module can now be used to configure Cumulus to
  integrate with a custom CMR host name and protocol (e.g. http://custom-cmr-host.com). Note
  that you must include a protocol (http:// or https://) if specifying a value for this
  variable.

All Changes

Added

• Added user doc describing new features related to the Cumulus dead letter archive.
• CUMULUS-2327
  • Added reserved concurrency setting to the Cumulus API lambda function.
  • Added relevant tfvars to the archive and cumulus terraform modules.
• CUMULUS-2460
  • Adds POST /executions/search-by-granules for retrieving executions from a list of granules or granule query
  • Adds searchExecutionsByGranules to @cumulus/api-client/executions
• CUMULUS-2475
  • Adds GET endpoint to distribution API
• CUMULUS-2463
  • PUT /granules reingest action allows a user to override the default execution
    to use by providing an optional workflowName or executionArn parameter on
    the request body.
  • PUT /granules/bulkReingest action allows a user to override the default
    execution/workflow combination to reingest with by providing an optional
    workflowName on the request body.
• Adds workflowName and executionArn params to @cumulus/api-client/reingestGranules
• CUMULUS-2476
  • Adds handler for authenticated HEAD Distribution requests replicating current behavior of TEA
• CUMULUS-2478
  • Implemented bucket map.
  • Implemented /locate endpoint
  • Cumulus distribution API checks the file request against bucket map:
    • retrieves the bucket and key from file path
    • determines if the file request is public based on the bucket map rather than the bucket type
    • (EDL only) restricts download from PRIVATE_BUCKETS to users who belong to certain EDL User Groups
    • bucket prefix and object prefix are supported
  • Add 'Bearer token' support as an authorization method
• CUMULUS-2486
  • Implemented support for custom headers
  • Added 'Bearer token' support as an authorization method
• CUMULUS-2487
  • Added integration test for cumulus distribution API
• CUMULUS-2569
  • Created bucket map cache for cumulus distribution API
• CUMULUS-2568
  • Add deletePdr/PDR deletion functionality to @cumulus/api-client/pdrs
  • Add removeCollectionAndAllDependencies to integration test helpers
  • Added example/spec/apiUtils.waitForApiStatus to wait for a
    record to be returned by the API with a specific value for
    status
  • Added example/spec/discoverUtils.uploadS3GranuleDataForDiscovery to upload granule data fixtures
    to S3 with a randomized granule ID for discover-granules based integration tests
  • Added example/spec/Collections.removeCollectionAndAllDependencies to remove a collection and
    all dependent objects (e.g. PDRs, granules, executions) from the
    database via the API
  • Added helpers to @cumulus/api-client:
    • pdrs.deletePdr - Delete a PDR via the API
    • replays.postKinesisReplays - Submit a POST request to the /replays endpoint for replaying Kinesis messages

Changed

• Moved functions from @cumulus/integration-tests to example/spec/helpers/workflowUtils:
  • startWorkflowExecution
  • startWorkflow
  • executeWorkflow
  • buildWorkflow
  • testWorkflow
  • buildAndExecuteWorkflow
  • buildAndStartWorkflow
• example/spec/helpers/workflowUtils.executeWorkflow now uses
  waitForApiStatus to ensure that the execution is completed or
  failed before resolving
• example/spec/helpers/testUtils.updateAndUploadTestFileToBucket
  now accepts an object of parameters rather than positional
  arguments
• Removed PDR from the payload in the input payload test fixture for reconciliation report integration tests
• The following integration tests for PDR-based workflows were
  updated to use randomized granule IDs:
  • example/spec/parallel/ingest/ingestFromPdrSpec.js
  • example/spec/parallel/ingest/ingestFromPdrWithChildWorkflowMetaSpec.js
  • example/spec/parallel/ingest/ingestFromPdrWithExecutionNamePrefixSpec.js
  • example/spec/parallel/ingest/ingestPdrWithNodeNameSpec.js
• Updated the @cumulus/api-client/CumulusApiClientError error class to include new properties that can be
  accessed directly on the error object:
  • statusCode - The HTTP status code of the API response
  • apiMessage - The message from the API response
• Added params.pRetryOptions parameter to
  @cumulus/api-client/granules.deleteGranule to control the retry
  behavior
• Updated cmr_custom_host variable to accept a full protocol and host name
  (e.g. http://cmr-custom-host.com), whereas it previously only accepted a host name
• CUMULUS-2482
  • Switches the default distribution app in the example/cumulus-tf deployment to the new Cumulus Distribution
  • TEA is still available by following instructions in example/README.md
• CUMULUS-2463
  • Increases the duration of allowed backoff times for a successful test from 0.5 sec to 1 sec.

Fixed

• Fixed bug where cmr_custom_host variable was not properly forwarded into archive, ingest, and
  sqs-message-remover modules from cumulus module
• Fixed bug where parse-pdr set a granule's provider to the entire provider record when a NODE_NAME
  is present. Expected behavior consistent with other tasks is to set the provider name in that field.
• CUMULUS-2568
  • Update reconciliation report integration test to have better cleanup/failure behavior
  • Fixed @cumulus/api-client/pdrs.getPdr to request correct endpoint for returning a PDR from the API
• CUMULUS-2620
  • Fixed a bug where a granule could be removed from CMR but still be set as
    published: true and with a CMR link in the Dynamo/PostgreSQL databases. Now,
    the CMR deletion and the Dynamo/PostgreSQL record updates will all succeed or fail
    together, preventing the database records from being out of sync with CMR.
  • Fixed @cumulus/api-client/pdrs.getPdr to request correct
    endpoint for returning a PDR from the API

v9.2.0

All Changes

Added

Cognito Distribution:

• CUMULUS-2475
  • Adds GET endpoint to distribution API
• CUMULUS-2476
  • Adds handler for authenticated HEAD Distribution requests replicating current behavior of TEA

Fixed

• CUMULUS-2520

  • Fixed error that prevented /elasticsearch/index-from-database from starting.

• CUMULUS-2558

  • Fixed issue where executions original_payload would not be retained on successful execution

v9.1.0

Breaking Changes

There are no migration steps associated with these changes.

• CUMULUS-2488
  • Removed all EMS reporting including lambdas, endpoints, params, etc as all
    reporting is now handled through Cloud Metrics

Breaking package changes

• CUMULUS-2472
  • Moved existing EarthdataLoginClient to @cumulus/oauth-client/EarthdataLoginClient and updated all references in Cumulus Core.
  • Rename EarthdataLoginClient property from earthdataLoginUrl to loginUrl for consistency with new OAuth clients. See example in oauth-client README

Notable Changes

• Can restrict direct S3 access to files based on CMR ACLs
• Fixed issue where the SyncGranule task did not use the correct key for checksum download

All Changes

Added

• CUMULUS-2354

  • Adds configuration options to allow /s3credentials endpoint to distribute
    same-region read-only tokens based on a user's CMR ACLs.
  • Configures the example deployment to enable this feature.

• CUMULUS-2442

  • Adds option to generate cloudfront URL to lzards-backup task. This will require a few new task config options that have been documented in the task README.

• CUMULUS-2497

  • Created isISOFile() to check if a CMR file is a CMR ISO file.

• CUMULUS-2371

  • Added helpers to @cumulus/ingest/sqs:
    • archiveSqsMessageToS3 - archives an incoming SQS message to S3
    • deleteArchivedMessageFromS3 - deletes a processed SQS message from S3
  • Added call to archiveSqsMessageToS3 to sqs-message-consumer which
    archives all incoming SQS messages to S3.
  • Added call to deleteArchivedMessageFrom to sqs-message-remover which
    deletes archived SQS message from S3 once it has been processed.

• Cumulus distribution with Cognito:

  • CUMULUS-2473
    • Updated tf-modules/cumulus_distribution module to take earthdata or cognito credentials
    • Configured example/cumulus-tf/cumulus_distribution.tf to use CSDAP credentials
  • CUMULUS-2474
    • Add S3ObjectStore to aws-client. This class allows for interaction with the S3 object store.
    • Add object-store package which contains abstracted object store functions for working with various cloud providers
  • CUMULUS-2470
    • Added /s3credentials endpoint for distribution API
  • CUMULUS-2477
    • Added /, /login and /logout endpoints to cumulus distribution API
  • CUMULUS-2479
    • Adds /version endpoint to distribution API

Changed

• HYRAX-439 - Corrected README.md according to a new Hyrax URL format.

• PR2224

  • Changed timeout on sfEventSqsToDbRecords Lambda to 60 seconds to match timeout for Knex library to acquire database connections

• CUMULUS-2208

  • Moved all @cumulus/api/es/* code to new @cumulus/es-client package
  • Changed timeout on sfEventSqsToDbRecords Lambda to 60 seconds to match timeout for Knex library to acquire database connections

• CUMULUS-2517

  • Updated postgres-migration-count-tool default concurrency to '1'

• CUMULUS-2489

  • Updated docs for Terraform references in FAQs, glossary, and in Deployment sections

• CUMULUS-2434

  • Updated @cumulus/cmrjs updateCMRMetadata and related functions to add
    both HTTPS URLS and S3 URIs to CMR metadata.
  • Updated update-granules-cmr-metadata-file-links task to add both HTTPS
    URLs and S3 URIs to the OnlineAccessURLs field of CMR metadata. The task
    configuration parameter cmrGranuleUrlType now has default value both.
  • To use the updated update-granules-cmr-metadata-file-links task, the
    granule UMM-G metadata should have version 1.6.2 or later, since CMR s3 link
    type 'GET DATA VIA DIRECT ACCESS' is not valid until UMM-G version
    1.6.2

• CUMULUS-2472

  • Renamed @cumulus/earthdata-login-client to more generic @cumulus/oauth-client as a parent class for new OAuth clients.
  • Added @cumulus/oauth-client/CognitoClient to interface with AWS cognito login service.

• CUMULUS-2497

  • Changed the @cumulus/cmrjs package:
    • Updated @cumulus/cmrjs/cmr-utils.getGranuleTemporalInfo() so it now
      returns temporal info for CMR ISO 19115 SMAP XML files.
    • Updated @cumulus/cmrjs/cmr-utils.isCmrFilename() to include
      isISOFile().

Fixed

• CUMULUS-2519

  • Update @cumulus/integration-tests.buildWorkflow to fail if provider/collection API response is not successful

• CUMULUS-2518

  • Update sf-event-sqs-to-db-records to not throw if a collection is not
    defined on a payload that has no granules/an empty granule payload object

• CUMULUS-2512

  • Updated ingest package S3 provider client to take additional parameter
    remoteAltBucket on download method to allow for per-file override of
    provider bucket for checksum
  • Updated @cumulus/ingest.fetchTextFile's signature to be parameterized and
    added remoteAltBucketto allow for an override of the passed in provider
    bucket for the source file
  • Update "eslint-plugin-import" to be pinned to 2.22.1

• CUMULUS-2520

  • Fixed error that prevented /elasticsearch/index-from-database from starting.

• 2231

  • Fixes broken relative path links in docs/README.md

Removed

• CUMULUS-2502
  • Removed outdated documenation regarding Kibana index patterns for metrics.

v8.1.1

Please note changes in 8.1.1 may not yet be released in future versions, as this
is a backport/patch release on the 8.x series of releases. Updates that are
included in the future will have a corresponding CHANGELOG entry in future releases.

Added

• CUMULUS-2497
  • Created isISOFile() to check if a CMR file is a CMR ISO file.

Fixed

• CUMULUS-2512
  • Updated ingest package S3 provider client to take additional parameter
    remoteAltBucket on download method to allow for per-file override of
    provider bucket for checksum
  • Updated @cumulus/ingest.fetchTextFile's signature to be parameterized and
    added remoteAltBucketto allow for an override of the passed in provider
    bucket for the source file
  • Update "eslint-plugin-import" to be pinned to 2.22.1

Changed

• CUMULUS-2497

  • Changed the @cumulus/cmrjs package:
    • Updated @cumulus/cmrjs/cmr-utils.getGranuleTemporalInfo() so it now
      returns temporal info for CMR ISO 19115 SMAP XML files.
    • Updated @cumulus/cmrjs/cmr-utils.isCmrFilename() to include
      isISOFile().

• 2216

  • Removed "node-forge", "xml-crypto" from audit whitelist, added "underscore"

v9.0.1

Cumulus v9.0.1 Migration steps

• This release of Cumulus enables integration with a PostgreSQL database for archiving Cumulus data. There are several upgrade steps involved, some of which need to be done before redeploying Cumulus. See the documentation on upgrading to the RDS release.

BREAKING CHANGES

• CUMULUS-2185 - RDS Migration Epic
  • CUMULUS-2191
    • Removed the following from the @cumulus/api/models.asyncOperation class in
      favor of the added @cumulus/async-operations module:
      • start
      • startAsyncOperations
  • CUMULUS-2187
    • The async-operations endpoint will now omit output instead of
      returning none when the operation did not return output.
  • CUMULUS-2309
    • Removed @cumulus/api/models/granule.unpublishAndDeleteGranule in favor
      of @cumulus/api/lib/granule-remove-from-cmr.unpublishGranule and
      @cumulus/api/lib/granule-delete.deleteGranuleAndFiles.
  • CUMULUS-2385
    • Updated sf-event-sqs-to-db-records to write a granule's files to
      PostgreSQL only after the workflow has exited the Running status.
      Please note that any workflow that uses sf_sqs_report_task for
      mid-workflow updates will be impacted.
    • Changed PostgreSQL file schema and TypeScript type definition to require
      bucket and key fields.
    • Updated granule/file write logic to mark a granule's status as "failed"
  • CUMULUS-2455
    • API move granule endpoint now moves granule files on a per-file basis
    • API move granule endpoint on granule file move failure will retain the
      file at it's original location, but continue to move any other granule
      files.
    • Removed the move method from the @cumulus/api/models.granule class.
      logic is now handled in @cumulus/api/endpoints/granules and is
      accessible via the Core API.

Added

• CUMULUS-2185 - RDS Migration Epic
  • CUMULUS-2130
    • Added postgres-migration-count-tool lambda/ECS task to allow for
      evaluation of database state
    • Added /migrationCounts api endpoint that allows running of the
      postgres-migration-count-tool as an asyncOperation
  • CUMULUS-2394
    • Updated PDR and Granule writes to check the step function
      workflow_start_time against the createdAt field for each record to ensure
      old records do not overwrite newer ones for legacy Dynamo and PostgreSQL
      writes
  • CUMULUS-2188
    • Added data-migration2 Lambda to be run after data-migration1
    • Added logic to data-migration2 Lambda for migrating execution records
      from DynamoDB to PostgreSQL
  • CUMULUS-2191
    • Added @cumulus/async-operations to core packages, exposing
      startAsyncOperation which will handle starting an async operation and
      adding an entry to both PostgreSQL and DynamoDb
  • CUMULUS-2127
    • Add schema migration for collections table
  • CUMULUS-2129
    • Added logic to data-migration1 Lambda for migrating collection records
      from Dynamo to PostgreSQL
  • CUMULUS-2157
    • Add schema migration for providers table
    • Added logic to data-migration1 Lambda for migrating provider records
      from Dynamo to PostgreSQL
  • CUMULUS-2187
    • Added logic to data-migration1 Lambda for migrating async operation
      records from Dynamo to PostgreSQL
  • CUMULUS-2198
    • Added logic to data-migration1 Lambda for migrating rule records from
      DynamoDB to PostgreSQL
  • CUMULUS-2182
    • Add schema migration for PDRs table
  • CUMULUS-2230
    • Add schema migration for rules table
  • CUMULUS-2183
    • Add schema migration for asyncOperations table
  • CUMULUS-2184
    • Add schema migration for executions table
  • CUMULUS-2257
    • Updated PostgreSQL table and column names to snake_case
    • Added translateApiAsyncOperationToPostgresAsyncOperation function to @cumulus/db
  • CUMULUS-2186
    • Added logic to data-migration2 Lambda for migrating PDR records from
      DynamoDB to PostgreSQL
  • CUMULUS-2235
    • Added initial ingest load spec test/utility
  • CUMULUS-2167
    • Added logic to data-migration2 Lambda for migrating Granule records from
      DynamoDB to PostgreSQL and parse Granule records to store File records in
      RDS.
  • CUMULUS-2367
    • Added granules_executions table to PostgreSQL schema to allow for a
      many-to-many relationship between granules and executions
      • The table refers to granule and execution records using foreign keys
        defined with ON CASCADE DELETE, which means that any time a granule or
        execution record is deleted, all of the records in the
        granules_executions table referring to that record will also be
        deleted.
    • Added upsertGranuleWithExecutionJoinRecord helper to @cumulus/db to
      allow for upserting a granule record and its corresponding
      granules_execution record
  • CUMULUS-2128
    • Added helper functions:
      • @cumulus/db/translate/file/translateApiFiletoPostgresFile
      • @cumulus/db/translate/file/translateApiGranuletoPostgresGranule
      • @cumulus/message/Providers/getMessageProvider
  • CUMULUS-2190
    • Added helper functions:
      • @cumulus/message/Executions/getMessageExecutionOriginalPayload
      • @cumulus/message/Executions/getMessageExecutionFinalPayload
      • @cumulus/message/workflows/getMessageWorkflowTasks
      • @cumulus/message/workflows/getMessageWorkflowStartTime
      • @cumulus/message/workflows/getMessageWorkflowStopTime
      • @cumulus/message/workflows/getMessageWorkflowName
  • CUMULUS-2192
    • Added helper functions:
      • @cumulus/message/PDRs/getMessagePdrRunningExecutions
      • @cumulus/message/PDRs/getMessagePdrCompletedExecutions
      • @cumulus/message/PDRs/getMessagePdrFailedExecutions
      • @cumulus/message/PDRs/getMessagePdrStats
      • @cumulus/message/PDRs/getPdrPercentCompletion
      • @cumulus/message/workflows/getWorkflowDuration
  • CUMULUS-2199
    • Added translateApiRuleToPostgresRule to @cumulus/db to translate API
      Rule to conform to Postgres Rule definition.
  • CUMUlUS-2128
    • Added "upsert" logic to the sfEventSqsToDbRecords Lambda for granule and
      file writes to the core PostgreSQL database
  • CUMULUS-2199
    • Updated Rules endpoint to write rules to core PostgreSQL database in
      addition to DynamoDB and to delete rules from the PostgreSQL database in
      addition to DynamoDB.
    • Updated create in Rules Model to take in optional createdAt parameter
      which sets the value of createdAt if not specified during function call.
  • CUMULUS-2189
    • Updated Provider endpoint logic to write providers in parallel to Core
      PostgreSQL database
    • Update integration tests to utilize API calls instead of direct
      api/model/Provider calls
  • CUMULUS-2191
    • Updated cumuluss/async-operation task to write async-operations to the
      PostgreSQL database.
  • CUMULUS-2228
    • Added logic to the sfEventSqsToDbRecords Lambda to write execution, PDR,
      and granule records to the core PostgreSQL database in parallel with
      writes to DynamoDB
  • CUMUlUS-2190
    • Added "upsert" logic to the sfEventSqsToDbRecords Lambda for PDR writes
      to the core PostgreSQL database
  • CUMUlUS-2192
    • Added "upsert" logic to the sfEventSqsToDbRecords Lambda for execution
      writes to the core PostgreSQL database
  • CUMULUS-2187
    • The async-operations endpoint will now omit output instead of
      returning none when the operation did not return output.
  • CUMULUS-2167
    • Change PostgreSQL schema definition for files to remove filename and
      name and only support file_name.
    • Change PostgreSQL schema definition for files to remove size to only
      support file_size.
    • Change PostgresFile to remove duplicate fields filename and name and
      rename size to file_size.
  • CUMULUS-2266
    • Change sf-event-sqs-to-db-records behavior to discard and not throw an
      error on an out-of-order/delayed message so as not to have it be sent to
      the DLQ.
  • CUMULUS-2305
    • Changed DELETE /pdrs/{pdrname} API behavior to also delete record from
      PostgreSQL database.
  • CUMULUS-2309
    • Changed DELETE /granules/{granuleName} API behavior to also delete
      record from PostgreSQL database.
    • Changed Bulk operation BULK_GRANULE_DELETE API behavior to also delete
      records from PostgreSQL database.
  • CUMULUS-2367
    • Updated granule_cumulus_id foreign key to granule in PostgreSQL files
      table to use a CASCADE delete, so records in the files table are
      automatically deleted by the database when the corresponding granule is
      deleted.
  • CUMULUS-2407
    • Updated data-migration1 and data-migration2 Lambdas to use UPSERT instead
      of UPDATE when migrating dynamoDB records to PostgreSQL.
    • Changed data-migration1 and data-migration2 logic to only update already
      migrated records if the incoming record update has a newer timestamp
  • CUMULUS-2329
    • Add write-db-dlq-records-to-s3 lambda.
    • Add terraform config to automatically write db records DLQ messages to an
      s3 archive on the system bucket.
    • Add unit tests and a component spec test for the above.
  • CUMULUS-2380
    • Add process-dead-letter-archive lambda to pick up and process dead letters in the S3 system bucket dead letter archive.
    • Add `/deadLetterArchive/recoverCumulusMes...