[Webhook] Datastore creation deadlock

[cheina97]

Hi 👋

I'm encountering some issues installing Kamaji using ArgoCD.

I'm installing the version v1.0.0 with the following helm values.

etcd:
  deploy: false
datastore:
  enabled: true
  nameOverride: "default"
  driver: PostgreSQL
  endpoints:    
    [kamaji-datastore-rw.kamaji-system.svc:5432]
  basicAuth:
    usernameSecret:
      keyPath: username
      name: superuser-secret-kamaji-datastore-db
      namespace: kamaji-system
    passwordSecret:
      keyPath: password
      name: superuser-secret-kamaji-datastore-db
      namespace: kamaji-system
  tlsConfig:
    certificateAuthority:      
      certificate:
        name: kamaji-datastore-ca
        namespace: kamaji-system
        keyPath: ca.crt
      privateKey:
        name: kamaji-datastore-ca
        namespace: kamaji-system
        keyPath: ca.key
    clientCertificate:      
      certificate:
        name: kamaji-datastore-server
        namespace: kamaji-system
        keyPath: tls.crt
      privateKey:
        name: kamaji-datastore-server
        namespace: kamaji-system
        keyPath: tls.key

I've noticed that when ArgoCD tries to apply the Datastore manifest the kamaji pod have not started yet and the webhook server cannot respond.
The cause is this validation

kamaji/cmd/manager/cmd.go

Lines 83 to 85 in f4c0cec

	if err = datastoreutils.CheckExists(ctx, scheme, datastore); err != nil {
	return err
	}

which is performed before the webhook server startup. This means that the webhook cannot start before the Datastore resource has been created, but the Datastore cannot be applied since no webhook is running, creating a deadlock.

What I do not understand is how such a problem has never been discussed in issues (probably I've missed some issues) and how it has not ever been a problem for other users.
What am I missing?

In liqo.io we had some similar issues and the final solution was to isolate webhooks in a separated pod https://github.com/liqotech/liqo/blob/master/cmd/webhook/main.go

Thanks in advice

[cheina97]

I've also noticed that the webhook mutates tenantControlPlane resources and sets the default datastore in case the "datastore" property is void. Considering that the default datastore inside the TenantControlPlane controller is only used here https://github.com/cheina97/kamaji/blob/c2bb50933a415ed6e53d4198d98bc55f2fdda981/controllers/tenantcontrolplane_controller.go#L317-L335, do you think the check mentioned above is still necessary?

[prometherion]

That check has been removed with #597, you can skip it by using an edge OCI image.

[prometherion]

Too many things at the same time, and strictly connected.

We do have Kamaji adopters using ArgoCD and they're not relying on the v1.0.0 Helm Chart, but rather, they're using the stable charts we release behind an active subscription: this means you're encountering such an egg/chicken issue.

Why don't we solve that? The v1.0.0 branch is there to give Kamaji a spin in "traditional" environments where users interact directly with Helm, or better, FluxCD.

According to what are your plans, I would suggest:

1. installing from source
2. or, building your OCI chart for source and store it where you prefer (e.g.: GHCR)

An immutable and GitOps installation of Kamaji is expected for production-grade environments, and it's what we try to monetize with our offering of packaged artefacts ready to use.

[prometherion]

@cheina97 JFI, with #805 we now support installation with Helm Charts, although with no version pinning:

helm upgrade --install kamaji clastix/kamaji --namespace kamaji-system --create-namespace --set 'resources=null' --version 0.0.0+latest

It's useful for testing purposes, without the need of installing from scratch: this should solve your issue with the kamaji-etcd issue.

[cheina97]

Thanks @prometherion