Release 15.0.2 (#3349)

* backport PR * finalizing docs and changing remaining groups * 15.1.0 release * doc updates * PR feedback + testing changes * Revert "doc updates" This reverts commit 5037012. * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * changing version number * small fixes * small fix --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]>
nasa · Apr 26, 2023 · 60f9950 · 60f9950
1 parent fb73047
commit 60f9950
Show file tree

Hide file tree

Showing 107 changed files with 597 additions and 485 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,15 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ## Unreleased
 
+## [v15.0.2] 2023-04-25
+
+### Fixed
+
+- **CUMULUS-3120**
+  - Fixed a bug by adding in `default_log_retention_periods` and `cloudwatch_log_retention_periods` 
+  to Cumulus modules so they can be used during deployment for configuring cloudwatch retention periods
+  - Updated cloudwatch retention documentation to reflect the bugfix changes
+
 ## [v15.0.1] 2023-04-20
 
 ### Changed
@@ -6896,7 +6905,8 @@ Note: There was an issue publishing 1.12.0. Upgrade to 1.12.1.
 
 ## [v1.0.0] - 2018-02-23
 
-[unreleased]: https://github.com/nasa/cumulus/compare/v15.0.1...HEAD
+[unreleased]: https://github.com/nasa/cumulus/compare/v15.0.2...HEAD
+[v15.0.2]: https://github.com/nasa/cumulus/compare/v15.0.1...v15.0.2
 [v15.0.1]: https://github.com/nasa/cumulus/compare/v15.0.0...v15.0.1
 [v15.0.0]: https://github.com/nasa/cumulus/compare/v14.1.0...v15.0.0
 [v14.1.0]: https://github.com/nasa/cumulus/compare/v14.0.0...v14.1.0

diff --git a/docs/configuration/cloudwatch-retention.md b/docs/configuration/cloudwatch-retention.md
@@ -35,44 +35,68 @@ Changing the log retention policy in the AWS Management Console is a fairly simp
 
 ## Terraform
 
-The `cumulus` module exposes values for configuration of log retention for
-cloudwatch log groups (in days). A configurable map of `cloudwatch_log_retention_periods` currently supports the following variables:
-
-- cumulus-tf_egress_lambda_log_retention
-- archive_private_api_log_retention
-- archive_api_log_retention
-- archive_async_operation_log_retention
-- archive_granule_files_cache_updater_log_retention
-- archive_publish_executions_log_retention
-- archive_publish_granule_log_retention
-- archive_publish_pdrs_log_retention
-- archive_replay_sqs_messages_log_retention
-- cumulus_distribution_api_log_retention
-- cumulus_ecs_service_default_log_retention
-- ingest_discover_pdrs_task_log_retention
-- ingest_hyrax_metadata_updates_task_log_retention
-- ingest_parse_pdr_task_log_retention
-- ingest_post_to_cmr_task_log_retention
-- ingest_queue_pdrs_task_log_retention
-- ingest_queue_workflow_task_log_retention
-- ingest_sync_granule_task_log_retention
-- ingest_update_cmr_access_constraints_task_log_retention
-
-In order to configure this value for the cloudwatch log group, the variable for the retention period for the respective group should be in the form of:
-
-```hcl
-<cumulus_module>_<cloudwatch_log_group>_log_retention: <log_retention>
-  type = number
+Cumulus modules create cloudwatch log groups and manage log retention for a subset of lambdas and tasks. These log groups have a default log retention time, but, there are two optional variables which can be set to change the default retention period for all or specific Cumulus managed cloudwatch log groups through deployment. For cloudwatch log groups which are not managed by Cumulus modules, the retention period is indefinite or `Never Expire` by AWS, cloudwatch log configurations for all Cumulus lambdas and tasks will be added in a future release.
+
+There are optional variables that can be set during deployment of cumulus modules to configure
+the retention period (in days) of cloudwatch log groups for lambdas and tasks which the `cumulus`, `cumulus_distribution`, and `cumulus_ecs_service` modules supports (using the `cumulus` module as an example):
+
+```tf
+module "cumulus" {
+  # ... other variables
+  default_log_retention_days = var.default_log_retention_days
+  cloudwatch_log_retention_periods = var.cloudwatch_log_retention_periods
+}
+```
+
+By setting the below variables in `terraform.tfvars` and deploying, the cloudwatch log groups will be instantiated or updated with the new retention value.
+
+### default_log_retention_periods
+
+The variable `default_log_retention_days` can be configured in order to set the default log retention for all cloudwatch log groups managed by Cumulus in case a custom value isn't used. The log groups will use this value for their retention, and if this value is not set either, the retention will default to 30 days. For example, if a user would like their log groups of the Cumulus module to have a retention period of one year, deploy the respective modules with the variable in the example below.
+
+#### Example
+
+```tf
+default_log_retention_periods = 365
 ```
 
-An example, in the case of configuring the retention period for the `parse_pdr_task` `aws_cloudwatch_log_group`:
+### cloudwatch_log_retention_periods
+
+The retention period (in days) of cloudwatch log groups for specific lambdas and tasks can be set
+during deployment using the `cloudwatch_log_retention_periods` terraform map variable. In order to
+configure these values for respective cloudwatch log groups, uncomment the `cloudwatch_log_retention_periods` variable and add the retention values listed below corresponding to the group's retention you want to change. The following values are supported correlating to their lambda/task name, (i.e. "/aws/lambda/prefix-DiscoverPdrs" would have the retention variable "DiscoverPdrs" )
+
+- ApiEndpoints
+- AsyncOperationEcsLogs
+- DiscoverPdrs
+- DistributionApiEndpoints
+- EcsLogs
+- granuleFilesCacheUpdater
+- HyraxMetadataUpdates
+- ParsePdr
+- PostToCmr
+- PrivateApiLambda
+- publishExecutions
+- publishGranules
+- publishPdrs
+- QueuePdrs
+- QueueWorkflow
+- replaySqsMessages
+- SyncGranule
+- UpdateCmrAccessConstraints
+
+:::note
+
+`EcsLogs` is used for all cumulus_ecs_service tasks cloudwatch log groups
+
+:::
 
-### Example
+#### Example
 
 ```tf
 cloudwatch_log_retention_periods = {
-  ingest_parse_pdr_task_log_retention = 365
+  ParsePdr = 365
 }
 ```
 
-Additionally, the variable `default_log_retention_days` can be configured separately during deployment in order to set the default log retention for the cloudwatch log groups in case a custom value isn't used. The log groups will use this value for their retention value, and if this value is not set either, the retention will default to 30 days.
+The retention periods are the number of days you'd like to retain the logs in the specified log group for. There is a list of possible values available in the [aws logs documentation](https://docs.aws.amazon.com/cli/latest/reference/logs/put-retention-policy.html).
diff --git a/example/cumulus-tf/cumulus_distribution.tf b/example/cumulus-tf/cumulus_distribution.tf
@@ -40,4 +40,6 @@ module "cumulus_distribution" {
   system_bucket            = var.system_bucket
   tags                     = local.tags
   vpc_id                   = local.vpc_id
+  default_log_retention_days       = var.default_log_retention_days
+  cloudwatch_log_retention_periods = var.cloudwatch_log_retention_periods
 }
diff --git a/example/cumulus-tf/ecs_hello_world_workflow.tf b/example/cumulus-tf/ecs_hello_world_workflow.tf
@@ -22,6 +22,8 @@ module "hello_world_service" {
 
   cpu                = 400
   memory_reservation = 700
+  default_log_retention_days                     = var.default_log_retention_days
+  cloudwatch_log_retention_periods               = var.cloudwatch_log_retention_periods
 
   environment = {
     AWS_DEFAULT_REGION = data.aws_region.current.name

diff --git a/example/cumulus-tf/main.tf b/example/cumulus-tf/main.tf
@@ -159,6 +159,8 @@ module "cumulus" {
   es_index_shards                           = var.es_index_shards
 
   dynamo_tables = merge(data.terraform_remote_state.data_persistence.outputs.dynamo_tables, var.optional_dynamo_tables)
+  default_log_retention_days = var.default_log_retention_days
+  cloudwatch_log_retention_periods = var.cloudwatch_log_retention_periods
 
   # Archive API settings
   token_secret = var.token_secret

diff --git a/example/cumulus-tf/python_processing_workflow.tf b/example/cumulus-tf/python_processing_workflow.tf
@@ -10,7 +10,8 @@ data "aws_ecr_repository" "cumulus_test_ingest_process" {
 
 module "python_test_ingest_processing_service" {
   source = "../../tf-modules/cumulus_ecs_service"
-
+  default_log_retention_days                     = var.default_log_retention_days
+  cloudwatch_log_retention_periods               = var.cloudwatch_log_retention_periods
   prefix = var.prefix
   name   = "PythonTestIngestProcess"
   tags   = local.tags

diff --git a/example/cumulus-tf/terraform.tfvars.example b/example/cumulus-tf/terraform.tfvars.example
@@ -69,11 +69,11 @@ csdap_host_url = "csdap host url"
 deploy_cumulus_distribution = true
 
 # Optional, uncomment if needed, these variables are for configuring the cloudwatch log group's retention periods
+# default_log_retention_days = 30
 # cloudwatch_log_retention_periods = {
-#   <module>_<cloudwatch_log_group_name>_log_retention = 365
+#   <lambda function or task name> = 365
 # }
 #
-# default_log_retention_days = 30
 
 # Optional. Uncomment if using Cumulus Distribution.
 # toggle this after deployed to put the correct port in. (and hosts and config)

diff --git a/example/cumulus-tf/thin_egress_app.tf b/example/cumulus-tf/thin_egress_app.tf
@@ -63,7 +63,7 @@ resource "aws_cloudwatch_log_subscription_filter" "egress_api_gateway_log_subscr
 resource "aws_cloudwatch_log_group" "egress_lambda_log_group" {
   count             = (var.log_destination_arn != null) ? 1 : 0
   name              = "/aws/lambda/${module.thin_egress_app.egress_lambda_name}"
-  retention_in_days = lookup(var.cloudwatch_log_retention_periods, "cumulus-tf_egress_lambda_log_retention", var.default_log_retention_days)
+  retention_in_days = lookup(var.cloudwatch_log_retention_periods, "thin-egress-app-EgressLambda", var.default_log_retention_days)
   tags              = var.tags
 }
 

diff --git a/example/cumulus-tf/variables.tf b/example/cumulus-tf/variables.tf
@@ -1,11 +1,5 @@
 # Required
 
-variable "cloudwatch_log_retention_periods" {
-  type = map(number)
-  description = "number of days logs will be retained for the respective cloudwatch log group, in the form of <module>_<cloudwatch_log_group_name>_log_retention"
-  default = {}
-}
-
 variable "cmr_client_id" {
   type = string
 }
@@ -56,12 +50,6 @@ variable "csdap_host_url" {
   description = "The csdap host url"
 }
 
-variable "default_log_retention_days" {
-  type = number
-  default = 30
-  description = "default value that user chooses for their log retention periods"
-}
-
 variable "launchpad_api" {
   type    = string
   default = "launchpadApi"
@@ -440,3 +428,34 @@ variable "subnets_tag_name" {
   type = string
   default = "Private application us-east-1a *"
 }
+
+variable "cloudwatch_log_retention_periods" {
+  type = map(number)
+  description = "retention periods for the respective cloudwatch log group, these values will be used instead of default retention days"
+  default = {
+    thin-egress-app-EgressLambda = 7
+    ApiEndpoints = 7
+    AsyncOperationEcsLogs = 7
+    DiscoverPdrs = 7
+    DistributionApiEndpoints = 7
+    EcsLogs = 7
+    granuleFilesCacheUpdater = 7
+    HyraxMetadataUpdates = 7
+    ParsePdr = 7
+    PostToCmr = 7
+    PrivateApiLambda = 7
+    publishExecutions = 7
+    publishGranules = 7
+    QueuePdrs = 7
+    QueueWorkflow = 7
+    replaySqsMessages = 7
+    SyncGranule = 7
+    UpdateCmrAccessConstraints = 7
+  }
+}
+
+variable "default_log_retention_days" {
+  type = number
+  default = 14
+  description = "default value that user chooses for their log retention periods"
+}
diff --git a/example/lambdas/asyncOperations/package.json b/example/lambdas/asyncOperations/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-async-operations",
-  "version": "15.0.1",
+  "version": "15.0.2",
   "description": "AsyncOperations Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/ftpPopulateTestLambda/package.json b/example/lambdas/ftpPopulateTestLambda/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/ftp-populate-test-lambda",
-  "version": "15.0.1",
+  "version": "15.0.2",
   "description": "FTP Population Utility Lambda",
   "main": "index.js",
   "private": true,
@@ -19,12 +19,12 @@
     "access": "private"
   },
   "dependencies": {
-    "@cumulus/api": "15.0.1",
-    "@cumulus/api-client": "15.0.1",
-    "@cumulus/common": "15.0.1",
-    "@cumulus/integration-tests": "15.0.1",
-    "@cumulus/logger": "15.0.1",
-    "@cumulus/test-data": "15.0.1",
+    "@cumulus/api": "15.0.2",
+    "@cumulus/api-client": "15.0.2",
+    "@cumulus/common": "15.0.2",
+    "@cumulus/integration-tests": "15.0.2",
+    "@cumulus/logger": "15.0.2",
+    "@cumulus/test-data": "15.0.2",
     "aws-sdk": "^2.585.0",
     "fs-extra": "^9.0.0",
     "jsftp": "https://github.com/jkovarik/jsftp.git#add_288",

diff --git a/example/lambdas/lzardsClientTest/package.json b/example/lambdas/lzardsClientTest/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-lzards-api-lambda",
-  "version": "15.0.1",
+  "version": "15.0.2",
   "description": "LZARDS API Client Test Lambda",
   "private": true,
   "engines": {
@@ -20,7 +20,7 @@
   "author": "Cumulus Authors",
   "license": "Apache-2.0",
   "dependencies": {
-    "@cumulus/logger": "15.0.1",
-    "@cumulus/lzards-api-client": "15.0.1"
+    "@cumulus/logger": "15.0.2",
+    "@cumulus/lzards-api-client": "15.0.2"
   }
 }
diff --git a/example/lambdas/python-processing/package.json b/example/lambdas/python-processing/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-process-activity",
   "private": true,
-  "version": "15.0.1",
+  "version": "15.0.2",
   "description": "Python reference activity",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-activity",
   "repository": {

diff --git a/example/lambdas/python-reference-activity/package.json b/example/lambdas/python-reference-activity/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-reference-activity",
   "private": true,
-  "version": "15.0.1",
+  "version": "15.0.2",
   "description": "Python reference activity",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-activity",
   "repository": {

diff --git a/example/lambdas/python-reference-task/package.json b/example/lambdas/python-reference-task/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-reference-task",
   "private": true,
-  "version": "15.0.1",
+  "version": "15.0.2",
   "description": "Python reference task",
   "main": "index.js",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-task",

diff --git a/example/lambdas/s3AccessTest/package.json b/example/lambdas/s3AccessTest/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-s3-access",
-  "version": "15.0.1",
+  "version": "15.0.2",
   "description": "S3 Access Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/snsS3Test/package.json b/example/lambdas/snsS3Test/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-sns-s3",
-  "version": "15.0.1",
+  "version": "15.0.2",
   "description": "SNS to S3 Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/versionUpTest/package.json b/example/lambdas/versionUpTest/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-version-up",
-  "version": "15.0.1",
+  "version": "15.0.2",
   "description": "Version Up Test Lambda",
   "main": "index.js",
   "private": true,