feat: support cilium cni

[okozachenko1203]

fix #124

[okozachenko1203]

https://opendev.org/openstack/magnum/src/commit/6bb2c107ffe0b9c7a278856927d64bebde3d5c36/magnum/api/validation.py#L324
magnum api validator has the supported network driver list as a constant value.
There are 2 options to make this configurable.

• get the supported supported network_driver list from the conf file as the allowed network_driver list
• override validator at the magnum drivers, i.e. in mcapi driver, we can define the validator. It is reasonable because the driver is the implementor for conducting tasks

[okozachenko1203]

depends-on: https://review.opendev.org/c/openstack/magnum/+/904835

[mnaser]

@okozachenko1203 Maybe we can simplifying the upstream change to this only:

-    supported_network_drivers = ['flannel', 'calico']
+    supported_network_drivers = ['flannel', 'calico', 'cilium']

[okozachenko1203]

@okozachenko1203 Maybe we can simplifying the upstream change to this only:

-    supported_network_drivers = ['flannel', 'calico']
+    supported_network_drivers = ['flannel', 'calico', 'cilium']

sure, i made another patch along side
https://review.opendev.org/c/openstack/magnum/+/905427

[mnaser]

@okozachenko1203 - good news, it seems upstreamed has merged the fix to allow it

[okozachenko1203]

root@kube-csqp0-rmggl-5jjss:/home/ubuntu# kubectl get po -A -owide
NAMESPACE     NAME                                             READY   STATUS    RESTARTS   AGE     IP             NODE                                    NOMINATED NODE   READINESS GATES
kube-system   cilium-c47bm                                     1/1     Running   0          77s     10.0.0.139     kube-csqp0-default-worker-bhfkv-qhg8l   <none>           <none>
kube-system   cilium-operator-68d4bbdf56-6jqh6                 1/1     Running   0          91s     10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   cilium-operator-68d4bbdf56-fjrxw                 1/1     Running   0          91s     10.0.0.139     kube-csqp0-default-worker-bhfkv-qhg8l   <none>           <none>
kube-system   cilium-qjdb7                                     1/1     Running   0          91s     10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   coredns-5d78c9869d-8876d                         1/1     Running   0          2m20s   10.100.0.158   kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   coredns-5d78c9869d-nr8m7                         1/1     Running   0          2m20s   10.100.0.57    kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   csi-cinder-controllerplugin-cd7ffbdf9-rl2fx      6/6     Running   0          91s     10.100.1.185   kube-csqp0-default-worker-bhfkv-qhg8l   <none>           <none>
kube-system   csi-cinder-nodeplugin-8fv7r                      3/3     Running   0          77s     10.0.0.139     kube-csqp0-default-worker-bhfkv-qhg8l   <none>           <none>
kube-system   csi-cinder-nodeplugin-llvr2                      3/3     Running   0          91s     10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   csi-nfs-controller-54fb58b59f-mlbvp              3/3     Running   0          89s     10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   csi-nfs-node-bpwxr                               3/3     Running   0          77s     10.0.0.139     kube-csqp0-default-worker-bhfkv-qhg8l   <none>           <none>
kube-system   csi-nfs-node-g9sql                               3/3     Running   0          89s     10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   etcd-kube-csqp0-rmggl-5jjss                      1/1     Running   0          2m20s   10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   k8s-keystone-auth-v8m5c                          1/1     Running   0          52s     10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   kube-apiserver-kube-csqp0-rmggl-5jjss            1/1     Running   0          2m20s   10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   kube-controller-manager-kube-csqp0-rmggl-5jjss   1/1     Running   0          2m20s   10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   kube-proxy-4sxm9                                 1/1     Running   0          77s     10.0.0.139     kube-csqp0-default-worker-bhfkv-qhg8l   <none>           <none>
kube-system   kube-proxy-shsfz                                 1/1     Running   0          2m20s   10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   kube-scheduler-kube-csqp0-rmggl-5jjss            1/1     Running   0          2m20s   10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   openstack-cloud-controller-manager-clqxr         1/1     Running   0          62s     10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>
kube-system   openstack-manila-csi-controllerplugin-0          4/4     Running   0          90s     10.100.1.149   kube-csqp0-default-worker-bhfkv-qhg8l   <none>           <none>
kube-system   openstack-manila-csi-nodeplugin-5fwpt            2/2     Running   0          77s     10.0.0.139     kube-csqp0-default-worker-bhfkv-qhg8l   <none>           <none>
kube-system   openstack-manila-csi-nodeplugin-p9mgp            2/2     Running   0          90s     10.0.0.247     kube-csqp0-rmggl-5jjss                  <none>           <none>

[mnaser]

Can you rebase this but also add Zuul jobs to test for both Calico and Cilium please?

[mnaser]

@okozachenko1203 can you look into the failures?

[okozachenko1203]

Failed tests:
[It] [sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] [Conformance]
[It] [sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance]
[It] [sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly] [Conformance]

[okozachenko1203]

cilium/cilium#14287
rancher/rke2#3003
cilium/cilium#16179

[okozachenko1203]

@mnaser now 1 test case is failing in conformance test and it is an upstream issue cilium/cilium#14287

[mnaser]

@okozachenko1203 Can you try and see if we enable the kube-proxy replcement mode will pass all tests?

[okozachenko1203]

@okozachenko1203 Can you try and see if we enable the kube-proxy replcement(KPR) mode will pass all tests?

@mnaser this means we have to provision the capi cluster without kube-proxy first. Then, enable kube-proxy replacement mode in the cilium chart's values.
Ok, I can see CABPK supports this already kubernetes-sigs/cluster-api#5993
Btw, do we have to expose a new config param to reflect enablement of KPR mode?

[okozachenko1203]

following this guide, we need to specify kube api server ip and port as helm values. But I don't think we can get them in advance.
@mnaser what do you think?

[mnaser]

@okozachenko1203 hmm, that would be quite the challenge to do this in that case, because I think this is quite a little more involved than "just a small change of Helm values" to enable this option.

[mnaser]

@okozachenko1203 Can you see how difficult it might be to enable Cilium to make this change happen with kube-proxy replacement out of the box? I think we'll have to look into some sort of 'reconcilation' loop..

[okozachenko1203]

@okozachenko1203 Can you see how difficult it might be to enable Cilium to make this change happen with kube-proxy replacement out of the box? I think we'll have to look into some sort of 'reconcilation' loop..

yeah, provisioning the cluster without kube-proxy will be simple by leveraging skipPhase option.
Then we have to reapply resourceset configmap for cilium chart after fetching the cluster's kube-api ip and port, and i think this should be tricky

[mnaser]

cni-chaining-mode: portmap
enable-session-affinity: 'true'

@okozachenko1203 can you just use these instead and keep things as the normal system?

reference: cilium/cilium#14287 (comment)

[okozachenko1203]

cni-chaining-mode: portmap
enable-session-affinity: 'true'

@okozachenko1203 can you just use these instead and keep things as the normal system?

reference: cilium/cilium#14287 (comment)

with this portmap mode, sonobuoy passed for cilium cni