Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

VM not picking assigned IP through VR #10608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
rbaweja opened this issue Mar 22, 2025 · 29 comments
Closed

VM not picking assigned IP through VR #10608

rbaweja opened this issue Mar 22, 2025 · 29 comments

Comments

@rbaweja
Copy link

rbaweja commented Mar 22, 2025

problem

A VM deployed on Apache CloudStack (ACS) is unable to acquire a DHCP-assigned IP address from the Virtual Router (VR). The DHCP request is visible in the VR’s tcpdump, but the VM does not receive an IP. However, manually assigning an IP to the VM allows it to reach both the VR and the internet.

versions

Environment Details
• CloudStack Version: 4.20
• Hypervisor: KVM
• Virtual Router IP: 10.0.0.1/24
• Network Setup: Advanced network with VR acting as DHCP server

The steps to reproduce the bug

  1. VM sends a DHCP renew request.
  2. VR receives the request, as confirmed via tcpdump -i eth2 port 67 or port 68 -n.
  3. VR responds with a DHCP reply to the VM.
  4. The VM does not acquire the IP.
  5. Manually assigning an IP allows the VM to reach both VR and the internet.

What to do about it?

No response
@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented Mar 22, 2025

Thanks for opening your first issue here! Be sure to follow the issue template!

@weizhouapache
Copy link
Member

This may be same as #10436

@rbaweja
Copy link
Author

rbaweja commented Mar 22, 2025

This may be same as #10436

i saw this post earlier, but didn't get solution by that method..

Do you suggest that i post it there or continue here?

@DaanHoogland
Copy link
Contributor

@rbaweja , so the dhcp reply is sent. Can you check on the client logs if anything goes wrong there? I think in the case of #10436 the lease is not acquired at all, so this is a different issue it seems.

@rbaweja
Copy link
Author

rbaweja commented Mar 24, 2025

On VM:

Image

New VR Logs:

Tue Mar 18 09:43:11 PM UTC 2025 Starting guest services for kvm
Tue Mar 18 09:43:13 PM UTC 2025 acpiphp and pci_hotplug module already compiled in
Tue Mar 18 09:43:14 PM UTC 2025 Received a new non-empty cmdline file from qemu-guest-agent
Tue Mar 18 09:43:14 PM UTC 2025 Booting from cloudstack, remove old configuration files in /etc/cloudstack/
Tue Mar 18 09:43:14 PM UTC 2025 Applying iptables rules
Tue Mar 18 09:43:14 PM UTC 2025 Setting up interface: eth0
Tue Mar 18 09:43:14 PM UTC 2025 Set up route for management network: via local gateway: for device eth0 for hypervisor: kvm
Tue Mar 18 09:43:16 PM UTC 2025 Executing cloud-early-config
Tue Mar 18 09:43:16 PM UTC 2025 Scripts checksum detected: oldmd5=0e40239dbb8573201b0a80a411b03b12 newmd5=0e40239dbb8573201b0a80a411b03b12
Tue Mar 18 09:43:18 PM UTC 2025 Could not find patch file, retrying
Tue Mar 18 09:43:20 PM UTC 2025 Could not find patch file, retrying
Tue Mar 18 09:43:22 PM UTC 2025 Could not find patch file, retrying
Tue Mar 18 09:43:24 PM UTC 2025 Could not find patch file, retrying
Tue Mar 18 09:43:24 PM UTC 2025 Scripts checksum detected: oldmd5=0e40239dbb8573201b0a80a411b03b12 newmd5=82d3395ea6108bf4a1ceaefdffb7a3f9
Tue Mar 18 09:43:24 PM UTC 2025 Patched scripts using /var/cache/cloud/cloud-scripts.tgz
Tue Mar 18 09:43:24 PM UTC 2025 Bootstrapping systemvm appliance
...skipping...
2025-03-24 14:07:48,627 INFO Executing: ip link show eth2 | grep ' state '
2025-03-24 14:07:48,631 INFO Executing2: arping -c 1 -I eth2 -A -U -s 10.10.10.1 10.10.10.1
2025-03-24 14:07:48,632 INFO Adding route: dev eth2 table: Table_eth2 network: 10.10.10.0/24 if not present
2025-03-24 14:07:48,632 INFO Executing: ip route show 10.10.10.0/24 table Table_eth2 proto static
2025-03-24 14:07:48,636 INFO Executing: sudo ip route flush cache
2025-03-24 14:07:48,647 ERROR Not able to setup source-nat for a regular router yet
2025-03-24 14:07:48,647 INFO Not making dns publicly available
2025-03-24 14:07:48,649 INFO Executing: systemctl start cloud-password-server@10.10.10.1
2025-03-24 14:07:48,659 INFO Service cloud-password-server@10.10.10.1 start
2025-03-24 14:07:48,660 INFO Nothing to commit. The /etc/monitor.conf file did not change
2025-03-24 14:07:48,660 INFO Nothing to commit. The /etc/cron.d/process file did not change
2025-03-24 14:07:48,661 INFO Executing: systemctl stop conntrackd
2025-03-24 14:07:48,674 INFO Service conntrackd stop
2025-03-24 14:07:48,674 INFO Executing: systemctl stop keepalived
2025-03-24 14:07:48,688 INFO Service keepalived stop
2025-03-24 14:07:48,688 INFO Executing: mount

@DaanHoogland
Copy link
Contributor

@rbaweja, can you check /var/log/dnsmasq.log on the router for any errors? I thought an answer went out of the VR as per you initial description but on second read it seems only the request may have come in.

@rbaweja
Copy link
Author

rbaweja commented Mar 24, 2025

@rbaweja, can you check /var/log/dnsmasq.log on the router for any errors? I thought an answer went out of the VR as per you initial description but on second read it seems only the request may have come in.

the worst part is there are 2 routers and both have same issue, i tried with default 5.5 centos template but same

Mar 24 15:58:34 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:34 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:34 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01

@DaanHoogland
Copy link
Contributor

that log looks ok,

you said

The DHCP request is visible in the VR’s tcpdump, but the VM does not receive an IP

do you mean you see the request coming in, but no response going out from the VR using tcpdump?

@rbaweja
Copy link
Author

rbaweja commented Mar 24, 2025

as per response i see that the response is also tehre but VM is not receiving the response.

tcpdump for VR:

17:15:21.881789 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:21.881790 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:21.881790 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:21.881790 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:21.882263 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:21.882528 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:21.882773 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:21.883042 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:29.882073 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:29.882074 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:29.882074 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:29.882074 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:29.882559 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:29.882815 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:29.882950 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:29.883171 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:41.883434 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:41.883694 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:41.883694 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:41.883694 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:41.884023 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:41.884335 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:41.884541 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:41.884779 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:51.883875 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:51.883876 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:51.884042 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:51.884042 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:51.884429 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:51.884779 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:51.885070 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:51.885342 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344

@weizhouapache
Copy link
Member

are there firewall rules in the vm ?

are there firewall rules with the kvm host ?

@rbaweja
Copy link
Author

rbaweja commented Mar 24, 2025

are there firewall rules in the vm ?

are there firewall rules with the kvm host ?

No firewall rules. Even the fresh VM of centos 5.5 doesn’t get IP

@weizhouapache
Copy link
Member

are there firewall rules in the vm ?

are there firewall rules with the kvm host ?

No firewall rules. Even the fresh VM of centos 5.5 doesn’t get IP

Are the vm and vr running on the same host ?

@rbaweja
Copy link
Author

rbaweja commented Mar 25, 2025

are there firewall rules in the vm ?
are there firewall rules with the kvm host ?

No firewall rules. Even the fresh VM of centos 5.5 doesn’t get IP

Are the vm and vr running on the same host ?

all on one host for now.

@weizhouapache
Copy link
Member

all on one host for now.

@rbaweja
can you restart network with cleanup and retry ?

please use tcpdump -vvv instead of tcpdump

@rbaweja
Copy link
Author

rbaweja commented Mar 25, 2025

tcpdump -vvv instead of tcpdump

this is what i get -

09:35:47.319133 IP (tos 0xc0, ttl 64, id 6910, offset 0, flags [none], proto UDP (17), length 372)
10.10.10.1.67 > 10.10.10.169.68: [udp sum ok] BOOTP/DHCP, Reply, length 344, xid 0xc11f8f51, secs 20, Flags [none] (0x0000)
Your-IP 10.10.10.169
Server-IP 10.10.10.1
Client-Ethernet-Address 02:01:00:cc:00:16
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
Server-ID (54), length 4: 10.10.10.1
Lease-Time (51), length 4: 4294967295
BR (28), length 4: 10.10.10.255
Hostname (12), length 39: "vm-d0d0ba5c-24e0-4acb-8cf0-785a1bef0e49"
Subnet-Mask (1), length 4: 255.255.255.0
Default-Gateway (3), length 4: 10.10.10.1
Domain-Name-Server (6), length 8: 10.10.10.1,172.27.5.11
Domain-Name (15), length 17: "cs2cloud.internal"
END (255), length 0
09:35:47.319469 IP (tos 0xc0, ttl 64, id 6911, offset 0, flags [none], proto UDP (17), length 372)
10.10.10.1.67 > 10.10.10.169.68: [udp sum ok] BOOTP/DHCP, Reply, length 344, xid 0xc11f8f51, secs 20, Flags [none] (0x0000)
Your-IP 10.10.10.169
Server-IP 10.10.10.1
Client-Ethernet-Address 02:01:00:cc:00:16
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
Server-ID (54), length 4: 10.10.10.1
Lease-Time (51), length 4: 4294967295
BR (28), length 4: 10.10.10.255
Hostname (12), length 39: "vm-d0d0ba5c-24e0-4acb-8cf0-785a1bef0e49"
Subnet-Mask (1), length 4: 255.255.255.0
Default-Gateway (3), length 4: 10.10.10.1
Domain-Name-Server (6), length 8: 10.10.10.1,172.27.5.11
Domain-Name (15), length 17: "cs2cloud.internal"
END (255), length 0

@weizhouapache
Copy link
Member

tcpdump -vvv instead of tcpdump

this is what i get -

09:35:47.319133 IP (tos 0xc0, ttl 64, id 6910, offset 0, flags [none], proto UDP (17), length 372) 10.10.10.1.67 > 10.10.10.169.68: [udp sum ok] BOOTP/DHCP, Reply, length 344, xid 0xc11f8f51, secs 20, Flags [none] (0x0000) Your-IP 10.10.10.169 Server-IP 10.10.10.1 Client-Ethernet-Address 02:01:00:cc:00:16 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message (53), length 1: Offer Server-ID (54), length 4: 10.10.10.1 Lease-Time (51), length 4: 4294967295 BR (28), length 4: 10.10.10.255 Hostname (12), length 39: "vm-d0d0ba5c-24e0-4acb-8cf0-785a1bef0e49" Subnet-Mask (1), length 4: 255.255.255.0 Default-Gateway (3), length 4: 10.10.10.1 Domain-Name-Server (6), length 8: 10.10.10.1,172.27.5.11 Domain-Name (15), length 17: "cs2cloud.internal" END (255), length 0 09:35:47.319469 IP (tos 0xc0, ttl 64, id 6911, offset 0, flags [none], proto UDP (17), length 372) 10.10.10.1.67 > 10.10.10.169.68: [udp sum ok] BOOTP/DHCP, Reply, length 344, xid 0xc11f8f51, secs 20, Flags [none] (0x0000) Your-IP 10.10.10.169 Server-IP 10.10.10.1 Client-Ethernet-Address 02:01:00:cc:00:16 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message (53), length 1: Offer Server-ID (54), length 4: 10.10.10.1 Lease-Time (51), length 4: 4294967295 BR (28), length 4: 10.10.10.255 Hostname (12), length 39: "vm-d0d0ba5c-24e0-4acb-8cf0-785a1bef0e49" Subnet-Mask (1), length 4: 255.255.255.0 Default-Gateway (3), length 4: 10.10.10.1 Domain-Name-Server (6), length 8: 10.10.10.1,172.27.5.11 Domain-Name (15), length 17: "cs2cloud.internal" END (255), length 0

is it the output inside the guest vm ?

@rbaweja
Copy link
Author

rbaweja commented Mar 25, 2025

this is from VR

@weizhouapache
Copy link
Member

VR looks ok

any packets received in the vm ?

@rbaweja
Copy link
Author

rbaweja commented Mar 25, 2025

VR looks ok

any packets received in the vm ?

Image (on VM side)

@weizhouapache
Copy link
Member

VR looks ok
any packets received in the vm ?

Image (on VM side)

it looks like no packets received.

do you use linux bridge or openvswitch ?

if you use linux bridge, can you capture the packets of the linux bridge br<devicename>-<vlanid> and the physical interface <devicename>.<vlanid> ?

@rbaweja
Copy link
Author

rbaweja commented Mar 25, 2025

VR looks ok
any packets received in the vm ?

Image (on VM side)

it looks like no packets received.

do you use linux bridge or openvswitch ?

if you use linux bridge, can you capture the packets of the linux bridge br<devicename>-<vlanid> and the physical interface <devicename>.<vlanid> ?

that also seems to be passing

10:42:23.421152 IP (tos 0xc0, ttl 64, id 11870, offset 0, flags [none], proto UDP (17), length 372)
10.10.10.1.67 > 10.10.10.169.68: [udp sum ok] BOOTP/DHCP, Reply, length 344, xid 0xf4d7fe4b, secs 12, Flags [none] (0x0000)
Your-IP 10.10.10.169
Server-IP 10.10.10.1
Client-Ethernet-Address 02:01:00:cc:00:16
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
Server-ID (54), length 4: 10.10.10.1
Lease-Time (51), length 4: 4294967295
BR (28), length 4: 10.10.10.255
Hostname (12), length 39: "vm-d0d0ba5c-24e0-4acb-8cf0-785a1bef0e49"
Subnet-Mask (1), length 4: 255.255.255.0
Default-Gateway (3), length 4: 10.10.10.1
Domain-Name-Server (6), length 8: 10.10.10.1,172.27.5.11
Domain-Name (15), length 17: "cs2cloud.internal"
10:42:23.599399 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 84a7.00:2a:6a:06:7b:c1.8098, length 42
message-age 0.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
root-id 84a7.00:2a:6a:06:7b:c1, root-pathcost 0, port-role Designated
10:42:23.599491 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 84a7.00:2a:6a:06:7b:c1.8097, length 42
message-age 0.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
root-id 84a7.00:2a:6a:06:7b:c1, root-pathcost 0, port-role Designated
10:42:24.527682 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 84a7.00:2a:6a:9c:98:fc.809b, length 42
message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
root-id 84a7.00:2a:6a:06:7b:c1, root-pathcost 2, port-role Designated
10:42:24.527816 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 84a7.00:2a:6a:9c:98:fc.8098, length 42
message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
root-id 84a7.00:2a:6a:06:7b:c1, root-pathcost 2, port-role Designated
10:42:25.646244 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 84a7.00:2a:6a:06:7b:c1.8098, length 42
message-age 0.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
root-id 84a7.00:2a:6a:06:7b:c1, root-pathcost 0, port-role Designated
10:42:25.646370 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 84a7.00:2a:6a:06:7b:c1.8097, length 42
message-age 0.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
root-id 84a7.00:2a:6a:06:7b:c1, root-pathcost 0, port-role Designated
10:42:26.568470 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 84a7.00:2a:6a:9c:98:fc.809b, length 42
message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
root-id 84a7.00:2a:6a:06:7b:c1, root-pathcost 2, port-role Designated
10:42:26.568640 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 84a7.00:2a:6a:9c:98:fc.8098, length 42
message-age 1.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s
root-id 84a7.00:2a:6a:06:7b:c1, root-pathcost 2, port-role Designated

@rbaweja
Copy link
Author

rbaweja commented Mar 25, 2025

explaining the scenario:
i have 3 network cards on my KVM
Cloudbr0 - Access mode for management
Cloudbr1 - guest network in trunk mode
cloudbr2 - public network in trunk mode.

i was using it with above scenario.

now i tried with all different versions but nothing worked. so i used cloudbr0 for both management and guest the VM got the IP.

since cloudbr0 is in access mode the traffic of other VLAN will not go to other hosts since it will not communicate.

now my question is how to keep all 3 networks isolated? shall i configure cloudbr1 with management vlan in native and rest as trunk ? or all should be like that?

@DaanHoogland
Copy link
Contributor

@rbaweja @weizhouapache , could this be a matter of asymetric routing? (just having a wild stab at it)

@rbaweja
Copy link
Author

rbaweja commented Mar 26, 2025

@rbaweja @weizhouapache , could this be a matter of asymetric routing? (just having a wild stab at it)

Not really sure.

@weizhouapache
Copy link
Member

explaining the scenario: i have 3 network cards on my KVM Cloudbr0 - Access mode for management Cloudbr1 - guest network in trunk mode cloudbr2 - public network in trunk mode.

i was using it with above scenario.

now i tried with all different versions but nothing worked. so i used cloudbr0 for both management and guest the VM got the IP.

since cloudbr0 is in access mode the traffic of other VLAN will not go to other hosts since it will not communicate.

now my question is how to keep all 3 networks isolated? shall i configure cloudbr1 with management vlan in native and rest as trunk ? or all should be like that?

@rbaweja
have you set the network traffic label of physical networks ?

refer to "Choose which traffic types will be carried by the physical network" section on https://docs.cloudstack.apache.org/en/latest/installguide/configuration.html

@rbaweja
Copy link
Author

rbaweja commented Mar 27, 2025

yes the label and configuration is perfect.

below for your reference:

network:
version: 2
ethernets:
ens160: {} # MNGT Network
ens192: {} # GUEST Trunk
ens224: {} # PUBLIC Trunk
bridges:
cloudbr0:
addresses:
- 172.27.5.31/24
dhcp4: false
routes:
- to: default
via: 172.27.5.1
nameservers:
addresses:
- 8.8.8.8
- 1.1.1.1
search: []
interfaces:
- ens160
parameters:
stp: false
forward-delay: 5
cloudbr1:
dhcp4: false
interfaces:
- ens192
cloudbr2:
dhcp4: false
interfaces:
- ens224
parameters:
stp: false
forward-delay: 5

/etc/cloudstack/agent/agent.properties
private.network.device=cloudbr0
guest.network.device=cloudbr1
hypervisor.type=kvm
port=8250
public.network.device=cloudbr2

@weizhouapache
Copy link
Member

yes the label and configuration is perfect.

below for your reference:

network: version: 2 ethernets: ens160: {} # MNGT Network ens192: {} # GUEST Trunk ens224: {} # PUBLIC Trunk bridges: cloudbr0: addresses: - 172.27.5.31/24 dhcp4: false routes: - to: default via: 172.27.5.1 nameservers: addresses: - 8.8.8.8 - 1.1.1.1 search: [] interfaces: - ens160 parameters: stp: false forward-delay: 5 cloudbr1: dhcp4: false interfaces: - ens192 cloudbr2: dhcp4: false interfaces: - ens224 parameters: stp: false forward-delay: 5

/etc/cloudstack/agent/agent.properties private.network.device=cloudbr0 guest.network.device=cloudbr1 hypervisor.type=kvm port=8250 public.network.device=cloudbr2

@rbaweja
can you check if the cloudbrX and its interface ensX have the same mac addresses ?

@rbaweja
Copy link
Author

rbaweja commented Mar 27, 2025

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr0 state UP group default qlen 1000
link/ether 00:50:56:95:74:67 brd ff:ff:ff:ff:ff:ff
altname enp3s0
3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr1 state UP group default qlen 1000
link/ether 00:50:56:95:58:d7 brd ff:ff:ff:ff:ff:ff
altname enp11s0
4: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr2 state UP group default qlen 1000
link/ether 00:50:56:95:31:af brd ff:ff:ff:ff:ff:ff
altname enp19s0
5: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 3e:f7:89:e8:5f:f3 brd ff:ff:ff:ff:ff:ff
inet 172.27.5.31/24 brd 172.27.5.255 scope global cloudbr0
valid_lft forever preferred_lft forever
inet6 fe80::3cf7:89ff:fee8:5ff3/64 scope link
valid_lft forever preferred_lft forever
6: cloudbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 42:6a:c2:33:d5:7e brd ff:ff:ff:ff:ff:ff
inet6 fe80::406a:c2ff:fe33:d57e/64 scope link
valid_lft forever preferred_lft forever
7: cloudbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether aa:fa:6d:75:e2:8d brd ff:ff:ff:ff:ff:ff
inet6 fe80::a8fa:6dff:fe75:e28d/64 scope link
valid_lft forever preferred_lft forever
8: cloud0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e2:ee:0e:49:fd:f7 brd ff:ff:ff:ff:ff:ff
inet 169.254.0.1/16 scope global cloud0
valid_lft forever preferred_lft forever
inet6 fe80::e0ee:eff:fe49:fdf7/64 scope link
valid_lft forever preferred_lft forever

@weizhouapache
Copy link
Member

@rbaweja
Can you set the mac address in netplan so that the linux bridge and ethernet use the same mac address ?

for example

    bridges:
        cloudbr0:   <=== linux bridge
            dhcp4: yes
            interfaces:
                - eth0      <====ethernet 
            macaddress: 1e:00:47:00:04:68   <========mac address of the ethernet

@apache apache locked and limited conversation to collaborators Apr 10, 2025
@DaanHoogland DaanHoogland converted this issue into discussion #10693 Apr 10, 2025

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@DaanHoogland @weizhouapache @rbaweja