LB for multus

[chuhuutiennam]

I want to build LB for multus network. I build and run loxiLB external(external: 4.0.5.2 and interanal: 4.0.6.149) and kube-loxilb(--loxiURL=http://4.0.6.149:11111) in cluster K8s. CNI using run for cluster multus macvlan(net1: 4.0.6.0/24). I create and run POD with eth0(10.x.x.x) and net1(4.0.6.3). But kube-loxilb run in cluster is ok but when I create service LB with name multus-service in K8s with link https://github.com/loxilb-io/kube-loxilb/tree/main/manifest/multus.
I have something not clear:

• k describe svc multus-service ==> Endpoint: 10.224.83.88:80 not 4.0.6.3:80
• docker exec -it loxilb loxilbcmd get lb => not create LB or endpoint
  
  
  
  
  

I don't know setup wrong something. I haven't lb in loxiLB external

[UltraInstinct14]

Double checking. Will confirm if there is any issue !!

[UltraInstinct14]

It is working fine in my test-bed. From the logs you shared in your test-setup, it looks like kube-loxilb is not getting service LB notification. You will get a bunch of logs in kube-loxilb when service related to loxilb is created . It can happen if loadbalancer class is not specified. For example as below -

apiVersion: v1
kind: Service
metadata:
  name: multus-service
  annotations:
    loxilb.io/multus-nets: macvlan1,macvlan2
    loxilb.io/staticIP: "4.0.5.2"
spec:
  externalTrafficPolicy: Local
  loadBalancerClass: loxilb.io/loxilb
  selector:
    app: pod-04
  ports:
    - port: 55002
      targetPort: 5002
  type: LoadBalancer

sudo kubectl describe svc multus-service

Name:                     multus-service
Namespace:                default
Labels:                   <none>
Annotations:              loxilb.io/multus-nets: macvlan1,macvlan2
                          loxilb.io/staticIP: 4.0.5.2
Selector:                 app=pod-04
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.96.141
IPs:                      10.43.96.141
LoadBalancer Ingress:     llb-4.0.5.2
Port:                     <unset>  55002/TCP
TargetPort:               5002/TCP
NodePort:                 <unset>  30687/TCP
Endpoints:                10.42.0.5:5002
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     32081
Events:                   <none>

Regarding the endpoint, kubectl would still show primary network endpoint. But loxilb can internally map to multus endpoint and if configured properly, loxicmd get lb -o wide will show the multus endpoints. I can debug further if you can share your exact yaml files for kube-loxilb, pod and multus-service creation. You can also join loxilb slack for quicker resolution and discussion.

[TrekkieCoder]

Hello All,

I have added a CICD to validate loxilb with multus based on this thread :

graph LR;
    Bastion-4.0.5.3-->4.0.5.2-LoxiLB-4.0.6.149;
    4.0.5.2-LoxiLB-4.0.6.149--multus-net-->4.0.6.150-K3S;

Loading

How to run (need vagrant) -

git clone https://github.com/loxilb-io/loxilb.git
cd loxilb/cicd/k3-flannel-multus
vagrant up

nginx service is automatically started -

Service Info
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
kubernetes       ClusterIP      10.43.0.1      <none>        443/TCP           3m3s
multus-service   LoadBalancer   10.43.151.40   llb-4.0.5.2   55002:31174/TCP   38s

LB info created in LoxiLB -

LB Info
| EXT IP  | SEC IPS | PORT  | PROTO |          NAME          | MARK | SEL |  MODE  | ENDPOINT | EPORT | WEIGHT | STATE | COUNTERS |
|---------|---------|-------|-------|------------------------|------|-----|--------|----------|-------|--------|-------|----------|
| 4.0.5.2 |         | 55002 | tcp   | default_multus-service |    0 | rr  | onearm | 4.0.6.3  |    80 |      1 | -     | 0:0      |
EP Info
|  HOST   |      NAME      | PTYPE | PORT | DURATION | RETRIES | MINDELAY | AVGDELAY | MAXDELAY | STATE |
|---------|----------------|-------|------|----------|---------|----------|----------|----------|-------|
| 4.0.6.3 | 4.0.6.3_tcp_80 | tcp:  |   80 |       60 |       2 |          |          |          | ok    |

You can run validation.sh script in the k3-flannel-multus dir or login to bastion host (vagrant ssh bastion) and curl to multus end-point. Based on this, I believe the OP can further debug his use-case. Meanwhile all working multus related config for this cicd scenario can be found here