Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple VMs Might Break Network Connections #36

Open
iwinux opened this issue Aug 7, 2023 · 4 comments
Open

Multiple VMs Might Break Network Connections #36

iwinux opened this issue Aug 7, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@iwinux
Copy link

iwinux commented Aug 7, 2023

OS / App Versions

  • macOS 13.5 (x86-64)
  • QEMU 7.2.1
  • socket_vmnet 1.1.2 built from source

Problem

The network of currently started QEMU VM stops working (symptom: ping 1.1.1.1 times out) when another VM (with unique MAC address) boots up but DOES NOT initiate any DHCP request.

However, after the second VM finishes its DHCP request, the network on first VM resumes to normal.

Steps to Reproduce

1. start socket_vmnet
sudo env DEBUG=true /usr/local/bin/socket_vmnet \
	--socket-group=admin \
	--vmnet-mode=shared \
	--vmnet-gateway=10.233.2.1 \
	--vmnet-dhcp-end=10.233.2.250 \
	--vmnet-mask=255.255.255.0 \
	/run/vmnet.sock
2. start VM `vmnet-test-01`
# Alpine Live ISOs available at https://www.alpinelinux.org/downloads/
# The "Virtual" ones are smallest

/usr/local/bin/socket_vmnet_client \
    /run/vmnet.sock \
    qemu-system-x86_64 \
        -name "vmnet-test-01" \
        -machine type=q35,accel=hvf \
        -cpu host \
        -smp cpus=1,sockets=1,cores=1,threads=1 \
        -m 1024 \
        -cdrom "$HOME/Downloads/alpine-virt-3.18.2-x86_64.iso" \
        -netdev socket,id=net0,fd=3 \
        -device "virtio-net-pci,netdev=net0,mac=de:ad:be:ef:00:01" \
        -object rng-random,filename=/dev/urandom,id=rng0 \
        -device virtio-rng-pci,rng=rng0 \
        -device virtio-gpu-pci \
        -nographic
3. acquire IP via DHCP for vmnet-test-01
# login with user "root"

cat > /etc/network/interfaces <<-END
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
END

rc-service networking start

# it pings successfully
ping -c 5 -w 1 1.1.1.1
4. start VM `vmnet-test-02`
/usr/local/bin/socket_vmnet_client \
    /run/vmnet.sock \
    qemu-system-x86_64 \
        -name "vmnet-test-02" \
        -machine type=q35,accel=hvf \
        -cpu host \
        -smp cpus=1,sockets=1,cores=1,threads=1 \
        -m 1024 \
        -cdrom "$HOME/Downloads/alpine-virt-3.18.2-x86_64.iso" \
        -netdev socket,id=net0,fd=3 \
        -device "virtio-net-pci,netdev=net0,mac=de:ad:be:ef:00:02" \
        -object rng-random,filename=/dev/urandom,id=rng0 \
        -device virtio-rng-pci,rng=rng0 \
        -device virtio-gpu-pci \
        -nographic
5. after 1-2 minutes, network on vmnet-test-01 stops working
# timeout
ping -c 5 -w 1 1.1.1.1

# DHCP cannot renew lease
rc-service networking restart
6. repeat step 3 for vmnet-test-02
# login with user "root"

cat > /etc/network/interfaces <<-END
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
END

rc-service networking start

# it pings successfully
ping -c 5 -w 1 1.1.1.1
7. surprisingly, now networks on both VMs resume to normal

Debug Log

https://gist.github.com/iwinux/305a9bea6b77c5c44494bb1883354f72

@AkihiroSuda AkihiroSuda added the bug Something isn't working label Aug 12, 2023
@nirs
Copy link
Member

nirs commented Nov 9, 2024

@iwinux can you reproduce this with lima >= 1.0.0?

Please test with current master, we have better logs now (see /var/log/socket_vmnet/stderr).

@nirs
Copy link
Member

nirs commented Nov 9, 2024

@iwinux can you reproduce this with lima >= 1.0.0?

I see you don't use lima, so it is not relevant.

@iwinux
Copy link
Author

iwinux commented Nov 11, 2024

@nirs Cool, just noticed a lot of new commits since September. Will try with latest master version later.

@nirs
Copy link
Member

nirs commented Nov 20, 2024

socket_vmnet send packets received from one vm or the vmnet interface to all other vms. It uses writev() which will block forever if the other side does not read from the socket.

Maybe qemu is not reading from the socket until the guest set up the network. In this case the socket buffer will become full at some point. This will cause socket_vment to block trying to send packet to the new vm.

It may help to ask about this in qemu-discuss mailing list.

If this is the root cause, it will be fixed by #77.

Another way to avoid this is to implement send timeout and drop the packet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants