disable-legacy-endpoints forces replacement

[brettcurtis]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.0.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v3.73.0
+ provider registry.terraform.io/hashicorp/google-beta v3.73.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.3.2
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

Affected Resource(s)

• google_container_cluster

Terraform Configuration Files

# Google Kubernetes Engine
# https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_cluster

resource "google_container_cluster" "this" {
  provider = google-beta

  project  = var.project_id
  name     = "${var.cluster_prefix}-${var.location}"
  location = var.location

  default_max_pods_per_node = var.max_pods_per_node
  min_master_version        = var.release_channel == "UNSPECIFIED" ? data.google_container_engine_versions.this.latest_master_version : null
  node_version              = var.release_channel == "UNSPECIFIED" ? data.google_container_engine_versions.this.latest_master_version : null
  network                   = "projects/${var.host_project}/global/networks/${var.network}"
  subnetwork                = "projects/${var.host_project}/regions/${var.subnet_region}/subnetworks/${var.subnet}"

  ip_allocation_policy {
    cluster_secondary_range_name  = var.cluster_secondary_range_name
    services_secondary_range_name = var.services_secondary_range_name
  }

  release_channel {
    channel = var.release_channel
  }

  addons_config {
    istio_config {
      disabled = !var.istio
    }
  }

  node_pool {
    name = "default-pool"
    node_config {
      machine_type = var.machine_type

      metadata = {
        "disable-legacy-endpoints" = true
      }

      oauth_scopes = [
        "https://www.googleapis.com/auth/cloud-platform",
      ]
    }

    initial_node_count = var.node_count

    autoscaling {
      min_node_count = var.min_node_count
      max_node_count = var.max_node_count
    }

    management {
      auto_repair  = "true"
      auto_upgrade = var.kubernetes_auto_upgrade
    }

    upgrade_settings {
      max_surge       = var.max_surge
      max_unavailable = var.max_unavailable
    }
  }

  maintenance_policy {
    daily_maintenance_window {
      start_time = var.kubernetes_daily_maintenance_window
    }
  }

  lifecycle {
    ignore_changes = [
      min_master_version,
      node_version,
    ]
  }

  depends_on = [
    google_project_service.this,
    google_compute_subnetwork_iam_member.service_network_gke_user,
    google_project_iam_member.host_service_agent,
  ]
}

Expected Behavior

Cluster should not rebuild.

Actual Behavior

          ~ node_config ***
              ~ disk_size_gb      = 100 -> (known after apply)
              ~ disk_type         = "pd-standard" -> (known after apply)
              ~ guest_accelerator = [] -> (known after apply)
              ~ image_type        = "COS_CONTAINERD" -> (known after apply)
              ~ labels            = *** -> (known after apply)
              ~ local_ssd_count   = 0 -> (known after apply)
              ~ metadata          = ***
                  - "disable-legacy-endpoints" = "true"
                *** -> (known after apply) # forces replacement

Steps to Reproduce

1. terraform apply

Important Factoids

References

• disable-legacy-endpoints forces new resource #3230

b/299312565

[venkykuberan]

I don't see that happening on my end. I tried the same tf version as yours. Can you please attach the debug log of the API response?

[brettcurtis]

plan.debug.gpg.txt

That what you're after? I set the TF_LOG=debug when running the plan. I encrypted using hashicorp public key.

[venkykuberan]

@brettcurtis can you please send the log in plain text, you can redact the project info.

[brettcurtis]

Haha, not really that log is FULL of stuff. Anything specific I can look for and share? Or another way to share the encrypted file. I was just following the notes in the issue template.

[gavinreaney]

I have the same issue.

In my case I am upgrading from terraform 0.13.7 to 1.0.4.

It seems that the previous state file has a metadata block, but we don't have that explicitly in our config.

            "node_config": [
              {
    ... snip ...
                "metadata": {
                  "disable-legacy-endpoints": "true"
                },

Somehow the switch to 1.0.4 is trying to remove that metadata block from the state. A workaround is probably for us to add this metadata explicitly to our config or perhaps an ignore block.

[brettcurtis]

Any update on this? We are seeing it across several clusters now, not sure what's going on exactly?

[brettcurtis]

@gavinreaney - my issue is slightly different in that I have the metadata block defined in our config since I hit this issue: #3230

[nstuart-idexx]

Also of note, going from 0.13 to any other later release (0.14,.15,1.x) triggers this behavior. I can run the same config on 0.13 and see no changes required, but any other version forces replacement with no changes to the config.

[kirkiris]

I had the same issue while upgrading from a very old version of the provider to the latest one (currently v3.89.0).

In your configuration, the value for disable-legacy-endpoints is type bool :

      metadata = {
        "disable-legacy-endpoints" = true
      }

I was able to overcome the issue by changing the type to string:

    metadata = {
      "disable-legacy-endpoints" = "true"
    }

Provider documentation is not very helpful, but I noticed that the referenced issue is being mentioned in this PR where, eventually, I saw this comment about the API expecting a string value. I tried it, it worked, I thought I'd share :)

[brettcurtis]

Interesting, thanks @kirkiris - should help us get by a few snags for sure!

[bgmonroe]

Is { "disable-legacy-endpoints" = "true" } even meaningful anymore?
According to this page, it sounds like the legacy metadata APIs were shutdown back in September 2020.