Extra state secret when using the kubernetes backend

[tculp]

What happened?

I have the terraform provider installed and configured to use kubernetes incluster-config as a state store. This seems to work great, and I get a state file for my Workspaces. However, once I apply a Workspace (terraform-test), I also get a state file secret that says it is from a non-existent default Workspace.

apiVersion: v1
data:
  tfstate: H4sIAAAAAAAA/0SMwa6CMBBF93xFM+tX8nAAW37FGDLgqMRKzbR1Q/h307pwec85uVulFLxZwuJXGFT7l3dkEbp6eY4/A03d1R0UH1gWcjCopk
y3rEw3zhEREpqj0Whx0r39R92ZA+rJXLi1vWUk+n74FF8pBhjUthcgHHySmTM6nQua7zw/RuGQXCnX5Fy1V58AAAD//5qciza0AAAA
kind: Secret
metadata:
  annotations:
    encoding: gzip
  creationTimestamp: "2023-11-09T20:22:32Z"
  labels:
    app.kubernetes.io/managed-by: terraform
    tfstate: "true"
    tfstateSecretSuffix: providerconfig-default
    tfstateWorkspace: default
  name: tfstate-default-providerconfig-default
  namespace: crossplane-system
  resourceVersion: "3127887"
  uid: 04bc13f5-9cc0-447f-81b4-bbea6baa164c
type: Opaque

This seems to happen the first time I apply a workspace with a new providerconfig.

tfstate-default-providerconfig-default                  Opaque                       1              38s
tfstate-default-providerconfig-defaulttwo               Opaque                       1              71s
tfstate-terraform-test-providerconfig-default              Opaque                       1              33s
tfstate-terraform-test-2-providerconfig-defaulttwo           Opaque                       1              66s

And if I delete my workspace, the tfstate-default-providerconfig-default secret remains.

How can we reproduce it?

aws-irsa DeploymentRuntimeConfig not included

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-terraform
spec:
  package: xpkg.upbound.io/upbound/provider-terraform:v0.11.0
  runtimeConfigRef:
    name: aws-irsa
---
apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    - filename: aws.json
      source: None

  configuration: |
    provider "aws" {}

    // Modules _must_ use remote state. The provider does not persist state.
    terraform {
      backend "kubernetes" {
        secret_suffix     = "providerconfig-default"
        namespace         = "crossplane-system"
        in_cluster_config = true
      }
    }
---
apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  name: terraform-test
spec:
  forProvider:
    source: Inline
    module: |
      data "aws_region"          "current" {}

      output "region" {
        value       = data.aws_region.current.name
        description = "The current AWS region."
      }

What environment did it happen in?

• Crossplane Version: 1.14.0
• Provider Version: 0.11.0
• Kubernetes Version: v1.28.3-eks-4f4795d
• Kubernetes Distribution: EKS

[bobh66]

I believe this is caused by the way the terraform CLI is implemented. If I remember correctly we have to run terraform init before we can set the specific workspace name - we can't set the workspace before we run terraform init. So that causes the default workspace to be created.

We don't use the default workspace for anything, it's just an artifact of the way the CLI works.

Is there a problem with having it there? I don't know that we can delete it permanently - it will come back every time the terraform CLI is executed.

[tculp]

It's not really a problem, just confusing and I don't think it's documented. I do wonder though what would happen if I named a workspace "default"

[bobh66]

I think it would just use the default workspace. The problem would be if you have more than one Workspace with the crossplane.io/external-name set to "default" because that's where we get the workspace name from. That's true for any external-name, since it's not unique it would cause collisions between Workspaces/workspaces. We should probably change that to use metadata.name instead so we are guaranteed a unique workspace name. @ytsarev any thoughts?

[tculp]

I did just try out making a workspace called default, and then making other workspaces and it didn't seem to cause an issue. I was worried it would delete the default workspace resources or something

[bobh66]

It will not try to delete the default workspace unless you use it as a workspace external-name, so in this case when you deleted the Workspace it also deleted the default workspace and then it was recreated the next time the terrform command was run.