Cloud Run it's recreating all the secret environment variables when adding/removing a secret env

[miguelangelmorenochacon]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

• Terraform v1.0.11
• google-beta ~> 3.90
• google ~>3.90

Affected Resource(s)

• google_cloud_run

Terraform Configuration Files

locals {
  /*
  * vpc-access-connector && static-outbound-ip  = "all"
  * vpc-access-connector only                   = "private-ranges-only"
  * otherwise	                                  = null
  */
  vpc_access_egress = var.enable_static_outbound_ip_address ? "all" : (var.vpc_access_connector_enabled ? "private-ranges-only" : null)
}

resource "google_cloud_run_service" "default" {
  name     = var.name
  location = var.location
  project  = var.project_id
  # FIXME: Use google provider when updating to ~>4.1
  provider = google-beta

  autogenerate_revision_name = true
  metadata {
    labels = merge(var.additional_labels)
  }
  template {
    metadata {
      annotations = {
        "run.googleapis.com/cloudsql-instances"   = var.cloudsql_instances
        "run.googleapis.com/vpc-access-connector" = var.vpc_access_connector_enabled ? google_vpc_access_connector.default[0].id : null
        "autoscaling.knative.dev/maxScale"        = var.max_scale
        "autoscaling.knative.dev/minScale"        = var.min_scale
        "run.googleapis.com/vpc-access-egress"    = local.vpc_access_egress
        "run.googleapis.com/secrets"              = var.secret_manager_project_id == null ? null : join(",", [for k, v in var.secret_environment_variables : "${k}:projects/${var.secret_manager_project_id}/secrets/${k}"])

        ## Dummy, only needed to be ignore
        "run.googleapis.com/client-name" = ""
        "run.googleapis.com/sandbox"     = ""
      }
    }
    spec {
      service_account_name  = local.service_account
      container_concurrency = var.container_concurrency
      timeout_seconds       = var.timeout_seconds
      containers {
        image   = var.container.image
        command = var.container.command
        args    = var.container.args

        dynamic "resources" {
          for_each = toset([var.container.resources])
          content {
            limits = {
              cpu    = resources.key["cpu"]
              memory = resources.key["memory"]
            }
          }
        }

        dynamic "ports" {
          for_each = var.container.port != null ? toset([var.container.port]) : []
          content {
            name           = "http1"
            container_port = ports.key
          }
        }

        dynamic "env" {
          for_each = var.container.envs == null ? {} : var.container.envs
          content {
            name  = env.key
            value = env.value
          }
        }

        dynamic "env" {
          for_each = var.secret_environment_variables
          content {
            name = env.value
            value_from {
              secret_key_ref {
                name = env.key
                key  = "latest"
              }
            }
          }
        }

      } # containers
    }   # spec
  }     # template

  lifecycle {
    ignore_changes = [
      metadata[0].annotations["run.googleapis.com/ingress"],
      metadata[0].annotations["client.knative.dev/user-image"],
      metadata[0].annotations["run.googleapis.com/client-name"],
      metadata[0].annotations["run.googleapis.com/client-version"],
      template[0].spec[0].containers[0].image,
      template[0].spec[0].containers[0].resources[0].requests,
      template[0].metadata[0].annotations["client.knative.dev/user-image"],
      template[0].metadata[0].annotations["run.googleapis.com/client-name"],
      template[0].metadata[0].annotations["run.googleapis.com/client-version"],
      template[0].metadata[0].annotations["run.googleapis.com/sandbox"],
      traffic,
    ]
  }
  depends_on = [
    google_secret_manager_secret_iam_member.member,
    google_secret_manager_secret_iam_member.member_with_external_sa,
  ]
}

Where var.secret_environment_variables takes this form

{
  "terraform-api-key-1"   = "API_KEY_1"
   "terraform-secret-1"    = "SECRET_1"
   "terraform-api-key-2" = "API_KEY_2"
}

and var.container.envs

 {
  "UPLOAD EVENTS"       = "sample-boolean"
  "DOMAIN"                      = "sample-domain"
  "APP_URL_TEMPLATE" = "sample-url"
}

Expected Behavior

Whe removing the "terraform-secret-1" = "SECRET_1" value from var.secret_environment_variables Cloud Run will need only to remove the environment variable relate dto that secret

  ~ resource "google_cloud_run_service" "default" {
        id                         = "locations/us-east1/namespaces/very-secret-project-id/services/sample-cloud-run-7928"
        name                       = "sample-cloud-run-7928"
        # (4 unchanged attributes hidden)


      ~ template {
          ~ metadata {
              ~ annotations = {
                  ~ "run.googleapis.com/secrets"              = "terraform-api-key-1:projects/very-secret-project-id/secrets/terraform-api-key-1,terraform-api-key-2:projects/very-secret-project-id/secrets/terraform-api-key-2,terraform-secret-1:projects/very-secret-project-id/secrets/terraform-secret-2" -> "terraform-api-key-1:projects/very-secret-project-id/secrets/terraform-api-key-1,terraform-api-key-2:projects/very-secret-project-id/secrets/terraform-api-key-2"
                    # (9 unchanged elements hidden)
                }
                name        = "sample-cloud-run-7928-e100a1bc-1637683284"
                # (2 unchanged attributes hidden)
            }

          ~ spec {
                # (3 unchanged attributes hidden)

              ~ containers {
                    # (3 unchanged attributes hidden)

                  - env {
                      - name = "SECRET_1" -> null

                      - value_from {
                          - secret_key_ref {
                              - key  = "latest" -> null
                              - name = "terraform-secret-1" -> null
                            }
                        }
                    }

                    # (40 unchanged blocks hidden)
                }
            }
        }

        # (2 unchanged blocks hidden)
    }

Actual Behavior

It needs to remove all the current secret env variables, and recreate them again except the one that is removed, that it's marked as removed

 # module.accounts.module.accounts-subscriber.google_cloud_run_service.default will be updated in-place
  ~ resource "google_cloud_run_service" "default" {
        id                         = "locations/us-east1/namespaces/very-secret-project-id/services/sample-cloud-run-7928"
        name                       = "sample-cloud-run-7928"
        # (4 unchanged attributes hidden)


      ~ template {
          ~ metadata {
              ~ annotations = {
                  ~ "run.googleapis.com/secrets"              = "terraform-api-key-1:projects/very-secret-project-id/secrets/terraform-api-key-1,terraform-api-key-2:projects/very-secret-project-id/secrets/terraform-api-key-2,terraform-secret-1:projects/very-secret-project-id/secrets/terraform-secret-2" -> "terraform-api-key-1:projects/very-secret-project-id/secrets/terraform-api-key-1,terraform-api-key-2:projects/very-secret-project-id/secrets/terraform-api-key-2"
                    # (9 unchanged elements hidden)
                }
                name        = "sample-cloud-run-7928-e100a1bc-1637683284"
                # (2 unchanged attributes hidden)
            }

          ~ spec {
                # (3 unchanged attributes hidden)

              ~ containers {
                    # (3 unchanged attributes hidden)

                  - env {
                      - name = "SECRET_1" -> null

                      - value_from {
                          - secret_key_ref {
                              - key  = "latest" -> null
                              - name = "terraform-secret-1" -> null
                            }
                        }
                    }
                  - env {
                      - name = "API_KEY_2" -> null

                      - value_from {
                          - secret_key_ref {
                              - key  = "latest" -> null
                              - name = "terraform-api-key-2" -> null
                            }
                        }
                    }
                  + env {
                      + name = "API_KEY_2"

                      + value_from {
                          + secret_key_ref {
                              + key  = "latest"
                              + name = "terraform-api-key-2"
                            }
                        }
                    }
                  + env {
                      + name = "API_KEY_1"

                      + value_from {
                          + secret_key_ref {
                              + key  = "latest"
                              + name = "terraform-api-key-1"
                            }
                        }
                    }
                  - env {
                      - name = "API_KEY_1" -> null

                      - value_from {
                          - secret_key_ref {
                              - key  = "latest" -> null
                              - name = "terraform-api-key-1" -> null
                            }
                        }
                    }


                    # (38 unchanged blocks hidden)
                }
            }
        }

        # (2 unchanged blocks hidden)
    }

Steps to Reproduce

1. Create a Cloud Run with multiple secret environment variables
2. Add or remove a new/existing environment variable

References

• b/240147716

[markesha]

I am facing the same during plans, not even adding or removing env vars.

[vgattuso]

I had the same issue with Terraform 1.3.3 and GCP provider 4.41.0. Only env var with secrets are affected by the delete/add issue.

It looks like the issue is in the Terraform SDK ApplyResourceChange method, the priorStateVal has value -> "" but the plannedStateVal has value -> nil, and the diff is detected. As a workaround, you can add value = "" in the TF config like in the example below. The expandCloudRunServiceSpecTemplateSpecContainersEnv func removes empty value fields so it's not sent to the GCP API.

env {
  name  = "ABC_SOME_SECRET_01"
  value = ""
  value_from {
    secret_key_ref {
      name = google_secret_manager_secret.some_secret_01.secret_id
      key  = "latest"
    }
  }
}

[jeremad]

for me it is fixed with It looks it is fixed by using https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_run_v2_service

[txomon]

I believe this is a duplicate of #9175

[Dragotic]

Any update here?

This is still an ongoing issue. Considering when you have 2 dynamic env blocks one for environment variables and one for environment secrets the following behavior takes place.

If you add a new variable in the first dynamic block, the environment variables, then the plan shows all the secrets from the second dynamic block being deleted and recreated.

I have:

env_vars  = {
  MY_ENV_VAR1 = MY_VALUE1
  MY_ENV_VAR2 = MY_VALUE2
}

env_secrets = ["MY_SECRET_KEY1", "MY_SECRET_KEY2"]

The env blocks look like this:

        dynamic "env" {
          for_each = var.env_vars
          content {
            name  = env.key
            value = env.value
          }
        }

        dynamic "env" {
          for_each = toset(var.env_secrets)

          content {
            name = env.key
            value_source {
              secret_key_ref {
                secret  = google_secret_manager_secret.this[env.key].secret_id
                version = "latest"
              }
            }
          }
        }

Adding a new env var like this

env_vars  = {
  MY_ENV_VAR1 = MY_VALUE1
  MY_ENV_VAR2 = MY_VALUE2
  MY_ENV_VAR3 = MY_VALUE3
}

causes changes where plan look like this:

  ~ resource "google_cloud_run_v2_service" "this" {
        id                      = "projects/project/locations/location/services/service"
        name                    = "service"
        # (2 7unchanged attributes hidden)

      ~ template {
            # (7 unchanged attributes hidden)

          ~ containers {
                name       = "service-1"
                # (4 unchanged attributes hidden)

              - env {
                  - name = "MY_SECRET_KEY1" -> null

                  - value_source {
                      - secret_key_ref {
                          - secret  = "MY_SECRET_KEY1" -> null
                          - version = "latest" -> null
                        }
                    }
                }
              - env {
                  - name = "MY_SECRET_KEY2" -> null

                  - value_source {
                      - secret_key_ref {
                          - secret  = "MY_SECRET_KEY2" -> null
                          - version = "latest" -> null
                        }
                    }
                }
              + env {
                  + name = "MY_SECRET_KEY1"

                  + value_source {
                      + secret_key_ref {
                          + secret  = "MY_SECRET_KEY1"
                          + version = "latest"
                        }
                    }
                }
              + env {
                  + name  = "MY_ENV_VAR3"
                  + value = "MY_VALUE3"
                }
              + env {
                  + name = "MY_SECRET_KEY2"

                  + value_source {
                      + secret_key_ref {
                          + secret  = "MY_SECRET_KEY2"
                          + version = "latest"
                        }
                    }
                }

                # (2 unchanged blocks hidden)
            }

            # (3 unchanged blocks hidden)
        }

        # (1 unchanged block hidden)
    }

[Pascal-Delange]

This is a serious pain