Is combination of kamaji and capi, compatible with kubevirt?

[cybercoder]

When i use kubeadm controlplane with capi and kubevirt, everything works fine, but not with kamaji controlplane replacement.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: testcluster
  namespace: "1731930192436"
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - 10.243.0.0/16
    services:
      cidrBlocks:
        - 10.95.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KamajiControlPlane
    name: testcluster
    namespace: "1731930192436"
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: KubevirtCluster
    name: testcluster
    namespace: "1731930192436"
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtCluster
metadata:
  name: testcluster
  namespace: "1731930192436"
# spec:
# controlPlaneServiceTemplate:
#   spec:
#     type: LoadBalancer
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtMachineTemplate
metadata:
  name: testcluster-control-plane
  namespace: "1731930192436"
spec:
  template:
    spec:
      virtualMachineBootstrapCheck:
        checkStrategy: ssh
      virtualMachineTemplate:
        metadata:
          namespace: "1731930192436"
        spec:
          runStrategy: Always
          template:
            spec:
              domain:
                cpu:
                  cores: 2
                devices:
                  disks:
                    - disk:
                        bus: virtio
                      name: containervolume
                  networkInterfaceMultiqueue: true
                memory:
                  guest: 4Gi
              evictionStrategy: External
              volumes:
                - containerDisk:
                    image: quay.io/capk/ubuntu-2204-container-disk:v1.30.1
                  name: containervolume
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
  name: testcluster
  namespace: "1731930192436"
spec:
  network:
    serviceType: LoadBalancer
  registry: registry.ik8s.ir
  version: v1.30.1
  dataStoreName: default
  # controlPlaneEndpoint:
  #   host: 172.16.16.225
  #   port: 6443
  apiServer:
    extraArgs:
      - --cloud-provider=external
  controllerManager:
    extraArgs:
      - --cloud-provider=external
  addons:
    coreDNS:
      dnsServiceIPs:
        - 10.95.0.10
    kubeProxy: {}
    konnectivity: {}
  kubelet:
    cgroupfs: systemd
    preferredAddressTypes:
      - InternalIP
      - ExternalIP
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtMachineTemplate
metadata:
  name: testcluster-md-0
  namespace: "1731930192436"
spec:
  template:
    spec:
      virtualMachineTemplate:
        metadata:
          annotations:
            username: "root"
            password: "toor"
          namespace: "1731930192436"
        spec:
          runStrategy: Always
          template:
            spec:
              domain:
                cpu:
                  cores: 2
                devices:
                  disks:
                    - disk:
                        bus: virtio
                      name: disk0
                memory:
                  guest: 4Gi
              volumes:
                - dataVolume:
                    name: disk0
                  name: disk0
              evictionStrategy: External
          dataVolumeTemplates:
            - apiVersion: "cdi.kubevirt.io/v1beta1"
              kind: DataVolume
              metadata:
                name: disk0
              spec:
                pvc:
                  accessModes:
                    - ReadWriteOnce
                  resources:
                    requests:
                      storage: 10Gi
                source:
                  http:
                    url: http://images.ik8s-system.svc.cluster.local/kube-noble-1.33.qcow2
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: testcluster-md-0
  namespace: "1731930192436"
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs: {}
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: testcluster-md-0
  namespace: "1731930192436"
spec:
  clusterName: testcluster
  replicas: 1
  selector:
    matchLabels: null
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: testcluster-md-0
          namespace: "1731930192436"
      clusterName: testcluster
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
        kind: KubevirtMachineTemplate
        name: testcluster-md-0
        namespace: "1731930192436"
      version: v1.30.1

And

k get svc -n 1731930192436
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
testcluster      LoadBalancer   10.43.212.56   172.16.16.225   6443:30517/TCP,8132:30143/TCP   6m42s
testcluster-lb   ClusterIP      10.43.111.99   <none>          6443/TCP                        6m42s

But there's no endpoint for testcluster-lb service!

k get endpoints -n 1731930192436 -w
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME             ENDPOINTS                                                        AGE
testcluster      10.16.1.145:6443,10.16.1.146:6443,10.16.1.145:8132 + 1 more...   6m58s
testcluster-lb   <none>                                                           6m58s

The worker vm shows this error after a awhile:

error execution phase preflight: couldn't validate the identity of the API Server:
failed to request the cluster-info ConfigMap ...
Get "https://10.43.111.99:6443/...": Client.Timeout exceeded

And kubelet service log:

journalctl -u kubelet -f

Sep 01 22:48:39 testcluster-md-0-dbrxj-h8tpg (kubelet)[5453]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_KUBEADM_ARGS
Sep 01 22:48:40 testcluster-md-0-dbrxj-h8tpg kubelet[5453]: E0901 22:48:40.005468    5453 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Sep 01 22:48:40 testcluster-md-0-dbrxj-h8tpg systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 01 22:48:40 testcluster-md-0-dbrxj-h8tpg systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 01 22:48:50 testcluster-md-0-dbrxj-h8tpg systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 50.
Sep 01 22:48:50 testcluster-md-0-dbrxj-h8tpg systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 01 22:48:50 testcluster-md-0-dbrxj-h8tpg (kubelet)[5567]: kubelet.service: Referenced but unset environment variable evaluates to an e

Also it shows it uses that service (without endpoint) inside the kubevirt vm:

cat /var/lib/cloud/instance/user-data.txt
## template: jinja
#cloud-config

write_files:
    - path: /run/kubeadm/kubeadm-join-config.yaml
      owner: root:root
      permissions: '0640'
      content: |
        ---
        apiVersion: kubeadm.k8s.io/v1beta3
        discovery:
          bootstrapToken:
            apiServerEndpoint: 10.43.111.99:6443
            caCertHashes:
            - sha256:c0e3c300b955a12083f3bdc7d536b407d9226dc6c9bdb7898644882a7d9048b7
            token: nj7lst.crzo36imbgsmj9am
        kind: JoinConfiguration
        nodeRegistration:
          imagePullPolicy: IfNotPresent
          taints:
          - effect: NoSchedule
            key: node.cluster.x-k8s.io/uninitialized
    - path: /run/cluster-api/placeholder
      owner: root:root
      permissions: '0640'
      content: "This placeholder file is used to create the /run/cluster-api sub directory in a way that is compatible with both Linux and Windows (mkdir -p /run/cluster-api does not work with Windows)"
runcmd:
    - kubeadm join --config /run/kubeadm/kubeadm-join-config.yaml  && echo success > /run/cluster-api/bootstrap-success.complete
users:
    - name: capk
      gecos: CAPK User
      sudo: ALL=(ALL) NOPASSWD:ALL
      groups: users, admin
      ssh_authorized_keys:
        - ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBIif6phvwjrAHZy8jwhGzljJaXoVZ2+UnJZCQzflIn2L+QW3tHDx1T8l1fmpO6SlgfIeMXyD21kT5scLxDDxUxHxHgZotAmjWV4c0xf07dtyKiFs1c8Ar9cCVOIqul5BGA==

[prometherion]

Your issue is with the KubevirtCluster resource:

https://github.com/clastix/cluster-api-control-plane-provider-kamaji/blob/master/docs/providers-kubevirt.md

Technical considerations

According to the said manifests, the KubevirtCluster resource must be marked as externally managed using the annotation cluster.x-k8s.io/managed-by.