Unable to customize OLM namespace

[jefflightweb]

What did you do?

Launched Helm install via OpenTofu/Terraform helm_release resource module, using customized set namespace and catalog_namespace parameters (both set to 'olm').

What did you expect to happen?

OLM and catalog to be installed in the 'olm' namespace.

What happened instead?

module.k8s-olm.kubernetes_namespace.this: Creating...
module.k8s-olm.helm_release.this: Creating...
module.k8s-olm.kubernetes_namespace.this: Creation complete after 1s [id=olm]

│ Error: Unable to continue with install: Namespace "olm" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "olm"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "olm"

The chart install succeeds if only catalog_namespace is set, but the main namespace defaults to 'operator-lifecycle-manager'.

If namespace is set {}, it always produces the above error.

Setting the namespace module variable appears to have no effect at all.

Your Environment

EKS 1.30

Teraform/Tofu code

locals {
  namespace     = "olm"
  chart_version = "0.28.0"
}

resource "kubernetes_namespace" "this" {
  metadata {
    name = local.namespace
  }
}

resource "helm_release" "this" {
  name       = local.namespace
  namespace  = local.namespace

  create_namespace = false

  repository = "oci://ghcr.io/cloudtooling/helm-charts"
  chart      = "olm"
  version    = local.chart_version
  set {
    name = "catalog_namespace"
    value = local.namespace
  }
  set {
    name = "namespace"
    value = local.namespace
  }

  wait       = true
}

[linear]

OSS-13 Unable to customize OLM namespace

[hypery2k]

I would suggest that you remove the namespace creation. The helm chart will create it for you.

Can you drop that resource and rerun?

[hypery2k]

mmmh strange. I‘ll check that

[jefflightweb]

I deleted my earlier comment so I could more thoroughly test. It takes awhile to create/destroy clusters on my end currently, and I wanted to make sure I was being completely accurate in my bug report.

I launched a fresh cluster to perform additional tests. I verified no 'olm' namespace existing in the cluster prior to each run.

1. I removed the namespace resource and changed create_namespace to 'true'. I get another error from the Helm chart install. I verified that the 'olm' namespace does indeed exist after the error is thrown. I have a feeling this is because I'm trying to set it for both catalog_namespace as well as namespace.

│ Error: 1 error occurred:
│ 	* namespaces "olm" already exists

2. I change the create_namespace setting to false. I make sure 'olm' namespace is not present in the cluster, I get the expected error of no namespace:

╷
│ Error: create: failed to create: namespaces "olm" not found
╵

[hypery2k]

ah ok. Got you now. For me this worked in:

locals {
  namespace     = "olm"
  chart_version = "0.28.0"
}

resource "helm_release" "this" {
  name       = local.namespace
  namespace  = local.namespace

  repository = "oci://ghcr.io/cloudtooling/helm-charts"
  chart      = "olm"
  create_namespace = true
  version    = local.chart_version
  set {
    name = "catalog_namespace"
    value = "olm"
  }

  wait       = true
}

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # helm_release.this will be created
  + resource "helm_release" "this" {
      + atomic                     = false
      + chart                      = "olm"
      + cleanup_on_fail            = false
      + create_namespace           = true
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "olm"
      + namespace                  = "olm"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "oci://ghcr.io/cloudtooling/helm-charts"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + verify                     = false
      + version                    = "0.28.0"
      + wait                       = true
      + wait_for_jobs              = false

      + set {
          + name  = "catalog_namespace"
          + value = "olm"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

helm_release.this: Creating...
helm_release.this: Still creating... [10s elapsed]
helm_release.this: Creation complete after 16s [id=olm]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

[jefflightweb]

Did you check the actual pods?

  # module.k8s-olm.helm_release.this will be created
  + resource "helm_release" "this" {
      + atomic                     = false
      + chart                      = "olm"
      + cleanup_on_fail            = false
      + create_namespace           = true
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "olm"
      + namespace                  = "olm"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "oci://ghcr.io/cloudtooling/helm-charts"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + verify                     = false
      + version                    = "0.28.0"
      + wait                       = true
      + wait_for_jobs              = false

      + set {
          + name  = "catalog_namespace"
          + value = "olm"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

olm                          operatorhubio-catalog-x464r                          1/1     Running   0          43s
operator-lifecycle-manager   catalog-operator-6c79dfd474-lbvqm                    1/1     Running   0          49s
operator-lifecycle-manager   olm-operator-6767dd8bdc-k2vbw                        1/1     Running   0          49s
operator-lifecycle-manager   packageserver-788dbfbb7f-kfsrj                       1/1     Running   0          43s
operator-lifecycle-manager   packageserver-788dbfbb7f-pz7vj                       1/1     Running   0          43s

Mine still defaults to 'operator-lifecycle-manager' namespace. I used the same code as you did.

[hypery2k]

mmmh, seems either a OLM or Helm issue. I found that I have to use .Release.Namespace instead of .Value.namespace but still getting errors:

helm_release.this: Creating...
╷
│ Warning: Helm release "" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│
│   with helm_release.this,
│   on main.tf line 20, in resource "helm_release" "this":
│   20: resource "helm_release" "this" {
│
╵
╷
│ Error: 2 errors occurred:
│ 	* namespaces "olm" already exists
│ 	* namespaces "olm-catalog" not found
│
│
│
│   with helm_release.this,
│   on main.tf line 20, in resource "helm_release" "this":
│   20: resource "helm_release" "this" {

[jefflightweb]

It would probably be fine to leave catalog_namespace as a default, but not being able to customize the main olm namespace just doesn't help our use-case at all...

[hypery2k]

okay, from chart side I cannot completly resolve the issue, due to kubernetes-sigs/cluster-api-addon-provider-helm#221

But here is how resolved the issue:

1. Install chart via helm: helm -n olm upgrade --install olm oci://ghcr.io/cloudtooling/helm-charts --version=0.28.1-dev --create-namespace --set catalog_namespace=olm. This will fail with:

Release "olm" does not exist. Installing it now.
Error: 1 error occurred:
	* namespaces "olm" already exists

2. Running it again will succeed: `

Release "olm" has been upgraded. Happy Helming!
NAME: olm
LAST DEPLOYED: Sat Oct  5 06:45:41 2024
NAMESPACE: olm
STATUS: deployed
REVISION: 2
TEST SUITE: None

3. Then import the chart to Terraform/OpenTofu: terraform import helm_release.this olm/olm
4. Running terraform apply succeds and the following namespaces are created:

olm                  Active   80s
operators            Active   80s

olm is used for pods:

kubectl -n olm get pod
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-857f7d5d7d-5db75   1/1     Running   0          9m1s
olm-operator-79978fc967-bsdkp       1/1     Running   0          9m1s
operatorhubio-catalog-56z7z         1/1     Running   0          8m54s
packageserver-5d97688f54-7gqcr      1/1     Running   0          8m55s
packageserver-5d97688f54-zc5kd      1/1     Running   0          8m55s

operators is empty until operators are added, e.g. gitlab

[hypery2k]

verified with this terraform script:

locals {
  namespace     = "olm"
  namespace_catalog     = "olm-catalog"
  namespace_operator    = "olm-operator"
  chart_version = "0.28.1-dev"
}

resource "helm_release" "this" {
  name       = local.namespace
  namespace  = local.namespace

  repository = "oci://ghcr.io/cloudtooling/helm-charts"
  chart      = "olm"
  create_namespace = true
  version    = local.chart_version
  set {
    name = "catalog_namespace"
    value = local.namespace
  }
  set {
    name = "operator_namespace"
    value = local.namespace_operator
  }

  wait       = true
}

[jefflightweb]

I ran the updated version, with your same Terraform code, and got a namespace error again, though it did install most of the components into the correct namespace now.

│ Warning: Helm release "" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
...
│ Error: 1 error occurred:
│ 	* namespaces "olm" already exists

I verified this twice with fresh clusters in between.

olm                  catalog-operator-6c79dfd474-9tvxj                    1/1     Running   0          98s
olm                  olm-operator-6767dd8bdc-k2xxg                        1/1     Running   0          98s
olm                  operatorhubio-catalog-btdnk                          1/1     Running   0          93s
olm                  packageserver-7bc6cdffbf-6l2wx                       1/1     Running   0          93s
olm                  packageserver-7bc6cdffbf-9nz4z                       1/1     Running   0          93s

locals {
  namespace     = "olm"
  namespace_catalog     = "olm-catalog"
  namespace_operator    = "olm-operator"
  chart_version = "0.28.1-dev"
}

resource "helm_release" "this" {
  name       = local.namespace
  namespace  = local.namespace

  repository = "oci://ghcr.io/cloudtooling/helm-charts"
  chart      = "olm"
  create_namespace = true
  version    = local.chart_version
  set {
    name = "catalog_namespace"
    value = local.namespace
  }
  set {
    name = "operator_namespace"
    value = local.namespace_operator
  }

  wait       = true
}

[jefflightweb]

I tried bringing back in the namespace creation via external resource, and then using create_namespace = false but it resulted in the same error before. If left to true, the chart installation always fails with the above error.

I seem to be stuck here, unfortunately.

[jefflightweb]

When I go to destroy a stack with that failed helm_release job, I run into another error and then have to expend effort in troubleshooting cluster operational status. It's a bit annoying unfortunately.

│ Error: Unable to continue with install: Namespace "olm" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "olm"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "olm"

[hypery2k]

mmmh I'm sorry. This is a little helm issue there as linked. I'll try to fix that otherwise.

Will take a look next week. Currently busy with other stuff

[jefflightweb]

Any update on this, by chance?

[hypery2k]

actually I cannot solve it via terraform and helm combination. The only option would be with the manual setup step. Currently it's now possible to resolve due to helm restrictions. Sorry to say that. In the current way olm works you either just use helm for templating with your custom values and apply it via kubectl or stick to the default values.
No other options currently left. Sorry