CKS Enhancements - Custom CNI Documentation

[jdhirst]

problem

I have been recently testing out the CKS enhancements in Cloudstack and have been wondering if anyone happens to have an example CNI configuration for Cilium?

Additionally, the Calico example in the documentation doesnt appear to work (at least with the 1.33 prebuilt ISO I was using)

Calico example here:
https://docs.cloudstack.apache.org/en/latest/plugins/cloudstack-kubernetes-service.html

ISO I used:
https://download.cloudstack.org/cks/setup-v1.33.1-calico-x86_64.iso
(I have also tested by building a custom kubernetes image following the documentation with the same results)

I am also using the default CloudStack 4.21 system VM instead of a custom node image.

When using the custom CNI config, I get the following error in the control node's cloud-init-output.log:

2025-09-20 06:44:36,835 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 431 column 4: "while scanning for the next token
found character '%' that cannot start any token
  in "<unicode string>", line 431, column 4:
      {% if registry is defined %}
       ^"
2025-09-20 06:44:36,849 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 431 column 4: "while scanning for the next token
found character '%' that cannot start any token
  in "<unicode string>", line 431, column 4:
      {% if registry is defined %}

I pulled the userdata files from the control node in question and base64 decoded them:
userdata_0.txt
userdata_1.txt

I can see this if registry is defined thing, but am not sure whether that means something is not being templated correctly or not:

  {% if registry is defined %}
  - path: /opt/bin/setup-containerd
    permissions: '0755'
    owner: root:root
    content: |
      #!/bin/bash -e

      export registryConfig="\\        [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"{{registry.url.endpoint}}\"]\n \\         endpoint = [\"{{registry.url}}\"]"
      export registryCredentials="\\      [plugins.\"io.containerd.grpc.v1.cri\".registry.configs.\"{{registry.url.endpoint}}\".auth]\n\tusername = \"{{registry.username}}\" \n\tpassword = \"{{registry.password}}\" \n\tidentitytoken = \"{{registry.token}}\""

      echo "creating config file for containerd"
      containerd config default > /etc/containerd/config.toml
      sed  -i '/\[plugins."io.containerd.grpc.v1.cri".registry\]/a '"${registryCredentials}"'' /etc/containerd/config.toml
      sed  -i '/\[plugins."io.containerd.grpc.v1.cri".registry.mirrors\]/a '"${registryConfig}"'' /etc/containerd/config.toml

      echo "Restarting containerd service"
      systemctl daemon-reload
      systemctl restart containerd
  {% endif %}

versions

cloudstack-management 4.21.0.0-1
cloudstack-agent 4.21.0.0-1

The steps to reproduce the bug

1. Either create a new kubernetes binaries ISO or use this one
2. Import the example calico CNI configuration from the documentation:

#cloud-config
- for i in {1..3}; do curl https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml -o /home/cloud/calico.yaml && break || sleep 5; done
- until [ -f /home/cloud/success ]; do sleep 5; done
- echo "Kubectl apply file"
- for i in {1..3}; do sudo /opt/bin/kubectl create -f /home/cloud/calico.yaml && break || sleep 5; done
- export PATH=$PATH:/home/cloud
- |
cat << 'EOF' > /home/cloud/create-configs.sh
#!/bin/bash
cat << 'EOL' > /home/cloud/bgp-config.yaml
apiVersion: crd.projectcalico.org/v1
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Debug
asNumber: {{ AS_NUMBER }}
EOL
cat << 'EOL' > /home/cloud/bgp-peer.yaml
apiVersion: crd.projectcalico.org/v1
kind: BGPPeer
metadata:
name: bgp-peer-example
spec:
peerIP: {{ ds.meta_data.peer_ip_address }}
asNumber: {{ ds.meta_data.peer_as_number }}
EOL
EOF
- chmod +x /home/cloud/create-configs.sh
- /home/cloud/create-configs.sh
- for i in {1..3}; do sudo /opt/bin/kubectl apply -f /home/cloud/bgp-config.yaml && break || sleep 5; done
- for i in {1..3}; do sudo /opt/bin/kubectl apply -f /home/cloud/bgp-peer.yaml && break || sleep 5; done

3. Create a new cluster and define the custom CNI configuration. It will be stuck in Starting state forever and the control node will fail to render the cloud init scripts properly. To login to the node, use the debian user instead of cloud since it hasn't yet rebooted into the new user.
   ...

What to do about it?

Please provide a working example of a custom CNI configuration for CloudStack 4.21.0. Either calico or cilium (preferred since I am hoping to build clusters with cilium).