Cloudstack creates VLAN sub-interface on wrong interface on KVM

[cisco-abrandel]

problem

I have a fresh cloudstack setup using advanced networking (without security groups) in a zone, where the VLAN sub-interface is being created on the wrong interface on the KVM hosts.

On the KVM host, I have a single physical interface attached to a bridge, cloudbr0. I then have a VLAN sub-interface for management configured on the bridge, as the port is a trunk and all VLANs we want to be tagged.

cloudbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:50:56:92:98:5b  txqueuelen 1000  (Ethernet)
        RX packets 5479511  bytes 7977472947 (7.4 GiB)
        RX errors 0  dropped 1976  overruns 0  frame 0
        TX packets 192836  bytes 1912563651 (1.7 GiB)
        TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0

cloudbr0.3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet y.y.y.181  netmask 255.255.255.0  broadcast y.y.y.255
        inet6 x:x:x:x:x:x  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::c551:98a9:1cbf:9df  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:92:98:5b  txqueuelen 1000  (Ethernet)
        RX packets 320144  bytes 1258105503 (1.1 GiB)
        RX errors 0  dropped 18  overruns 0  frame 0
        TX packets 192837  bytes 1912563993 (1.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:50:56:92:98:5b  txqueuelen 1000  (Ethernet)
        RX packets 6535565  bytes 8146940376 (7.5 GiB)
        RX errors 0  dropped 66  overruns 0  frame 0
        TX packets 195594  bytes 1912679835 (1.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

This works fine, cloudstack can add the host, communication works as expected.

The problem is when I go and create either a guest network, or assign public IPs with a VLAN tag, the VLAN sub-interface along with a new bridge is getting created on ens33 instead of cloudbr0.

brens33-30: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:50:56:92:98:5b  txqueuelen 1000  (Ethernet)
        RX packets 3603  bytes 275466 (269.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2  bytes 108 (108.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:50:56:92:98:5b  txqueuelen 1000  (Ethernet)
        RX packets 6535565  bytes 8146940376 (7.5 GiB)
        RX errors 0  dropped 66  overruns 0  frame 0
        TX packets 195594  bytes 1912679835 (1.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33.30: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:50:56:92:98:5b  txqueuelen 1000  (Ethernet)
        RX packets 3615  bytes 276276 (269.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7  bytes 306 (306.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

I can't figure out why this is happening. The labels seem to be configured properly in the agent:

private.network.device=cloudbr0
guest.network.device=cloudbr0
public.network.device=cloudbr0

To describe a bit more about our use case, we're primarily looking to leverage L2 networks in cloudstack. Each host will have a single interface that is trunked, and we will be dividing traffic based on VLANs. I would have expected that cloudstack creates the VLAN sub-interfaces on the cloudbr0 bridge, and not the parent of this bridge.

versions

Cloudstack version: 4.20.2.0
Rocky Linux 9.6

The steps to reproduce the bug

1. Install cloudstack
2. Configure a zone with advanced networking, no security groups
3. Create a single physical network, with all traffic types
4. Assign KVM network label as "cloudbr0"
5. Wait for system VMs to spin up (or spin your own) and watch cloudstack create the VLAN sub-interface on the wrong interface

What to do about it?

Cloudstack should use the right bridge

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[rajujith]

@cisco-abrandel could you try this bridge configuration on the KVM:

cloudbr0 -> ens33.3
cloudbr1 -> ens33

[cisco-abrandel]

Thanks, this got me a lot further. Will the system VMs always launch with eth1 attached to cloudbr0? It seems like no matter what I configure for the label or VLAN tag on the management traffic type, the system VMs always have eth1 against cloudbr0.

[rajujith]

As per the above example, you should be using cloudbr1 as the traffic label, I think, for guest and public. I didn't get the eth1 problem. Your host interface is en33 right? If it's about the eth1 of the SystemVMs, by design,eth1 is the management interface, and it will use the bridge as configured in the traffic label ( check the zone->physical networks), also check the agent properties. You can change the traffic label if it's not configured to use the desired bridge.

[weizhouapache]

this is intended I think.
cloudstack does create a virtual nic for vlan <physical nic name>.<vlan>, not <bridge name>.<vlan>

have you encountered any problem caused by it ? @cisco-abrandel

[cisco-abrandel]

I was seeing what I thought were issues related to this, but it turns out the issues were elsewhere. For full disclosure, we're running some small KVM test hosts on VMware, and has promiscuous enabled for the port group. This caused some minor looping of packets, one side effect of which was the linux bridge learned the KVM guests MAC on the wrong port occasionally, causing traffic blackholing. The solution for our specific topology was to disable promiscuous on the VMware port group the KVM host are in, and instead use MAC learning on the DVS.