virtio-blk poll still causes VM_EXITs

[enterJazz]

so I finally got around to benchmarking virtio-blk poll correctly; while it does decrease VM_EXITs and increase performance, it still have a bunch of VM_EXITs. Here are my current results:

setup based off of https://github.com/finallyjustice/sample/blob/master/performance/virtio-blk-poll/virtio-blk-poll.txt

# w/ poll queues = 4

-- numjobs=1 : IOPS = 99.4K,  avg latency = 2356.4usec,  vm_exits = 4384787


# w/ poll queues = 2

-- numjobs=1 : IOPS = 98K,  avg latency = 2396.78usec,  vm_exits = 4656416


# w/o poll

-- numjobs=1 : IOPS = 96.1K,  avg latency = 244.41usec,  vm_exits = 4814550

I will go ahead and ask in the qemu matrix channel if we can get rid of VM_EXITs using virtio-blk poll- if I am lucky, I may get a good response.

[enterJazz]

Here is my question:

Hi, I have a question regarding virtio-blk polling.
I'm currently trying to do polling-based storage IO, and thus, avoid VM_EXITs.
I went ahead and recreated the virtio-blk polling benchmarks ( from https://lkml.org/lkml/2021/5/20/2047 and https://github.com/finallyjustice/sample/blob/master/performance/virtio-blk-poll/virtio-blk-poll.txt );
while I found that the poll queues do increase IOPS and average latency, I still get a high number of VM_EXITs.
The poll queues do decrease the number of VM_EXITs, however, I had hoped that I could eliminate the exits, as mentioned in https://lkml.org/lkml/2021/6/3/697.
Specifically:

... QEMU already has virtqueue polling
code that disables the vq notification (the virtio-pci hardware register
write that causes a vmexit).

Any pointers on:

1. How to eliminate the VM_EXITs w/ virtio-blk polling?
2. What QEMU virtqueue polling code this message refers to?

Thanks!

===

Will update if I get a response

[enterJazz]

I got a response asking if the virtqueue kick is causing the vm exit, here is my respnose:

=======

I believe so, however, I am not entirely sure if I checked correctly.

Within the VM, I checked for the invocation of virtqueue_kick; IIRC kick means a write to the pci register, causing the VM_EXIT(?).
I checked via: bpftrace -e 'kprobe:virtqueue_kick { @[kstack]+=1 }'
I found that w/ and w/o polling have roughly the same number of invocations of virtqueue_kick (io_engine=pvsync2, w/ and w/o hipri).

I also traced kvm events from the host via tracecmd.
I found a large number of kvm_mmio events that look like this:
.qemu-system-x8-153389 [033] 491681.376371: kvm_mmio: mmio write len 4 gpa 0xfee00380 val 0x7fffffff
I assume these are the writes to the pci register (?).

Hence, it seems completion polling works fine, however, qemu-side submission polling is not taking effect?

[enterJazz]

I got this response:

stsquad
looking at the kick logic it seems vring_need_event() will always kick if the new index is above the last notified index, so from this I assume any rate limiting of kicks is controlled by the driver
indeed the virtio block driver does
if (bd->last && virtqueue_kick_prepare(vblk->vqs[qid].vq))
notify = true;
so only kicks at the end of a block of submissions, I don't think it has the fallback to polling of the network drivers...

[enterJazz]

I got some responses from stefanha 😍

enterjazz
: The poll queues do not avoid vmexit. They avoid host->guest interrupts.
enterjazz
: If you want to avoid vmexits, make sure the device keeps virtqueue kick notifications disabled.
enterjazz
: (that requires a device implementation that polls the virtqueues to detect new requests).
https://docs.oasis-open.org/virtio/virtio/v1.2/csd01/virtio-v1.2-csd01.html#x1-8200010

[enterJazz]

Regarding stefanha's comment:

as he mentioned, the backend virtio device needs to keep kick notifications disabled. To achieve this, the device needs the Descriptor Ring Change Event Flags implemented as in https://docs.oasis-open.org/virtio/virtio/v1.2/csd01/virtio-v1.2-csd01.html#x1-8200010

Luckily, it seems that DPDK implements this:
https://github.com/DPDK/dpdk/blob/6ef07151aac4b4d9601d547f94a996d1c71b3871/drivers/net/virtio/virtio_ring.h#L76

IIRC correctly, SPDK uses DPDK for the data plane; I need to check on this, but maybe this is something you know @mmisono ?
Best case, SPDK polls virtqueues via DPDK to detect new requests.

So, ideally:

• we use SPDK backend
• SPDK backend device disables virtqueue kick notifications
• SPDK backend polls virtqueues to detect new requests
  --> no vm_exits

Now we only need to check if this approach works. Looks promising though!

[enterJazz]

Update: looks like SPDK virtio does polling! 🎉

The driver, just like the SPDK vhost Target, is using pollers instead of standard interrupts to check for an I/O response.

https://spdk.io/doc/virtio.html

Hence: next steps are to set up SPDK w/ polling, and check for VM_EXITs.
There may be some effort to get SPDK virtio working w/ SEV, however, it appears to be the correct way forwards.

[enterJazz]

Relevant for setup: https://spdk.io/doc/bdev.html#bdev_config_virtio_blk

[enterJazz]

I guess we could use vfio-user after all? https://github.com/spdk/spdk/blob/0f57273a578706b63281d885ff4c82e665719b76/test/vfio_user/virtio/initiator_bdevperf.sh#L26

@mmisono I remember there were issues setting up vfio-user-pci ; I guess we should not go down this route , and go for vhost-blk instead

although i'm pretty sure vhost-blk does not work out-of-the box either.
I'm currently checking if creating the virtio-blk endpoint somehow enables qemu to use it as a device without specifying chardev ( I think this created the issues )

[mmisono]

• As Stephan said, the host driver also needs to poll to avoid all vmexit
• iirc vfio-user has slightly better performance than vhost-blk. One important thing is vfio-user version uses (not virtio but) nvme interface, and Intel's security guidelines on confidential virtual machines only recommend to use virtio to harden security. So there is a trade off
• if spdk vhost-blk works, then it should be fine as a first step
• the reason vfio-user does not work with sev is libvfio-user requires to have access to the all guest memory using sharemem device but qemu does not allow it for sev machine. But as qemu is able to access shared memory, I believe it should be possible for vfio-user to have access to the shared mem as well. This would require qemu modification. I think we need to make qemu to pass mmaped shared memory file descriptor to libvfio-user process somehow.
• I guess vhost-blk also has a similar issue but maybe not ..

[mmisono]

Btw, we can quickly check if spdk vhost and vfio-user eliminate vmexit using normal VM