virtio-blk optimization #8612

shiqingg · 2024-05-28T08:23:33Z

No description provided.

ACRN_IOEVENTFD_FLAG_ASYNCIO is not set when unregister ioeventfd in the current implementation which will cause the old asyncio_desc will be remained in hypervisor link list when switching from OVMF to kernel. Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]>

ACRN virtio devices are using a per device mutex to protect the concurrent operations on the device's PIO/MMIO. This introduces big contention in fast IO hence downgrades the IO performance, for example virtio-blk with asyncio enabled. This patch introduces per queue mutex to relieve such issues. Currently the per queue mutex is only used in the asycio path when iothread is enabled. Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]>

Virtio legacy device (ver < 1.0) uses a single PIO for all virtqueues. Notifications from different virtqueues are implemented by writing virtqueue index to the PIO. Writing different values to the same addr needs to be mapped to different eventfds by asyncio. This is called data match feature of asyncio. v3 -> v4: * Update the definition of `struct asyncio_desc` Use `struct acrn_asyncio_info` inside it, instaed of defining the duplicated fileds. * Update `add_asyncio` to use `memcpy_s` rather than assigning all the fields using 5 assignment statements. * Update `asyncio_is_conflict` for coding style 120-character line is sufficient to write all conditions. * Update the checks related to `wildcard` Because we require every conditional clause to have a Boolean type in the coding guideline. v2 -> v3: No change v1 -> v2: No change Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]> Signed-off-by: Shiqing Gao <[email protected]>

Virtio-blk can support multiple virtqueues (mq) which is negotiated between FE and BE by the feature bit VIRTIO_BLK_F_MQ. The virtqueue number of virtio-blk can be specified by "mq=x" in the parameter. For example: "virtio-blk,iothread,mq=2,..." Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]>

block_if is the backend of ahci and virtio-blk. Only one queue is supported by block_if now. Several worker threads are created as the thread pool for the queue. One BIG mutex is used for the queue and thread operation. With this patch block_if can support multiple queues and each queue is backed by several worker threads. blockif_req can be submited/enqueued into one specified queue. By spliting into several queues contention from the BIG mutex can be relieved/eliminated. This is used to support virtio-blk multiple queues feature. Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]>

io_uring is a high-performance asynchronous I/O framework, primarily designed to improve the efficiency of input and output (I/O) operations in user-space applications. This patch enables io_uring in block_if module. It utilizes the interfaces provided by the user-space library `liburing` to interact with io_uring in kernel-space. To build the acrn-dm with io_uring support, `liburing-dev` package needs to be installed. For example, it can be installed like below in Ubuntu 22.04. sudo apt install liburing-dev In order to support both the thread pool mechanism and the io_uring mechanism, an acrn-dm option `aio` is introduced. By default, thread pool mechanism is selected. - Example to use io_uring: `add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback,aio=io_uring` - Example to use thread pool: `add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback,aio=threads` - Example to use thread pool (by default): `add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback` v2 -> v3: * Update iothread_handler - Use the unified eventfd interfaces to read the counter value of the ioeventfd. - Remove the while loop to read the ioeventfd. It is not necessary because one read would reset the counter value to 0. * Update iou_submit_sqe to return an error code The caller of iou_submit_sqe shall check the return value. If there is NO available submission queue entry in the submission queue, need to break the while loop. Request can only be submitted when SQE is available. v1 -> v2: * move the logic of reading out ioeventfd from iothread.c to virtio.c, because it is specific to the virtqueue handling. Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]>

Prior to this patch, one single iothread instance is created and initialized in the `main` function. This single iothread monitors all the registered fds and handles all the corresponding requests. It leads to the limited flexibility of the iothread support. To improve the flexibility of the iothread support, this patch does: - add the support of multiple iothread instances. `iothread_create` is introduced to create a certain number of iothread instances. It shall be called at first by each virtual device owner (such as virtio-blk BE) on initialization phase. Then, `iothread_add` can be called to add the to be monitored fd to the specified iothread. - update virtio-blk BE to let the acrn-dm option `iothread` accept a number as the number of iothread instances to be created. If `iothread` is contained in the parameters, but the number is not specified, one iothread instance would be created by default. Examples to specify the number of iothread instances: 1. Create 2 iothread instances `add_virtual_device 9 virtio-blk iothread=2,mq=2,/dev/nvme1n1,writeback,aio=io_uring` 2. Create 1 iothread instances (by default) `add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback,aio=io_uring` - update virtio-blk BE to separate the request handling of different virtqueues to different iothreads. The request from one or more virtqueues can be handled in one iothread. The mapping between virtqueues and iothreads is based on round robin. v1 -> v2: * add a mutex to protect the free ioctx slot allocation Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]>

To improve the performance of the virtual device who utilizes iothread (such as virtio-blk), this patch sets iothread nice value to PRIO_MIN, so that it could get higher priority on scheduling. This patch does: - introduce `set_thread_priority` to set the priority of the current running thread. The priority could be any value in the range PRIO_MIN to PRIO_MAX. Lower numerical value causes more favorable scheduling. - set iothread nice value to PRIO_MIN. Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]> Signed-off-by: Shiqing Gao <[email protected]>

This patch adds an acrn-dm option `nocache` to bypass the Service VM's page cache. - By default, the Service VM's page cache is utilized. - If `nocache` is specified in acrn-dm parameters, the Service VM's page cache would be bypassed (opening the file/block with O_DIRECT flag). Example to bypass the Service VM's page cache: `add_virtual_device 5 virtio-blk iothread,mq=2,/dev/nvme2n1,writeback,nocache` Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]>

Use of O_DIRECT flag could be a performance option. But this flag may impose alignment restrictions on the length and address of user-space buffers and the file offset of I/Os. To support the use of O_DIRECT flag in block_if, this patch adds the support to handle the misaligned request. - When O_DIRECT flag is used (`nocache` is specified in acrn-dm parameters), * if the original I/O request is aligned, the original I/O request is submitted directly. * if the original I/O request is not aligned (either due to the buffer address/length misalignment, or the offset misalignment), the misaligned request is converted to an aligned request before submission. - When O_DIRECT flag is not used, the original I/O request is submitted directly. v1 -> v2: * cleanup the free() logic in `blockif_init_bounced_write` Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]>

This patch renames the iothread for better readability. For instance, the new name of the iothread for virtio-blk device looks like `iothr-0-blk9:0`. It could be helpful when tuning the performance and the CPU utilization. v1 -> v2: * add `const` qualifier for the input parameter of `iothread_create` Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]>

This patch updates the `iothread` option to specify the CPU affinity of the iothread. Setting the iothread's CPU affinity could benefit the Service VM's CPU utilization when Service VM owns limited dedicated CPUs. It could be helpful to ensure the I/O mediator Quality of Service (QoS). Once the performance tuning is done, the specific CPU affinity config could pass to acrn-dm directly, letting the deployment more easily. The format looks like below: iothread=<num_iothread>@<cpu_affinity> "@" is used to separate the following two settings: - the number of iothread instances - the CPU affinity settings for each iothread instance. The format of `cpu_affinity` looks like below: <cpu_affinity_0>/<cpu_affinity_1>/<cpu_affinity_2>/... 1. "/" is used to separate the CPU affinity setting for each iothread instance (sequentially). 2. char '*' can be used to skip the setting for the specific iothread instance. 3. the number of cpu_affinity_x vs. the number of iothread instances - If # of cpu_affinity_x is less than # of iothread instances, no CPU affinity settings for the last few iothread instances. - If # of cpu_affinity_x is more than # of iothread instances, the extra cpu_affinity_x are discarded. 4. ":" is used to separate different CPU cores for each CPU affinity setting. Examples to specify the CPU affinity of the iothread: 1. iothread=3@0:1:2/0:1 `add_virtual_device 9 virtio-blk iothread=3@0:1:2/0:1,mq=3,/dev/nvme1n1` a) 3 iothread instances are created. b) CPU affinity of iothread instances for this virtio-blk device: - 1st iothread instance <-> pins to Service VM CPU 0,1,2 - 2nd iothread instance <-> pins to Service VM CPU 0,1 - 3rd iothread instance <-> No CPU affinity settings 2. iothread=3@0/*/1 `add_virtual_device 9 virtio-blk iothread=3@0/*/1,mq=3,/dev/nvme1n1` a) 3 iothread instances are created. b) CPU affinity of iothread instances for this virtio-blk device: - 1st iothread instance <-> pins to Service VM CPU 0 - 2nd iothread instance <-> No CPU affinity settings - 3rd iothread instance <-> pins to Service VM CPU 1 v1 -> v2: * encapsulate one API in iothread.c to parse the iothread options, so that other BE can also use it. v2 -> v3: * introduce one API iothread_free_options to free the elements that are allocated dynamically in iothread_parse_options(). Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]>

When `io_uring` is used, `blockif_flush_cache` is missing when an WRITE operation is completed. `blockif_flush_cache` would flush the modified in-core data to the disk device according to the setting of the cache mode. Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]>

When multiple virtio-blk instances are created for one VM, using the same `static struct virtio_ops virtio_blk_ops` for all instances is buggy. It only works when all instances are created with the same number of the virtqueues. This patch fixes this issue by introducing a member in `struct virtio_blk` to store the ops info for each virtio-blk instance. Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]>

With current implementation, in blockif_dequeue/blockif_complete, if the current request is consecutive to any request in penq or busyq, current request's status is set to BST_BLOCK. Then, this request is blocked until the prior request, which blocks it, is completed. It indicates that consecutive requests are executed sequentially. This patch adds a flag `no_bst_block` to bypass such logic because: 1. the benefit of this logic is not noticeable; 2. there is a chance that a request is enqueued in block_if_queue but not dequeued when this logic is triggered along with the io_uring mechanism; Example to use this flag: `add_virtual_device 5 virtio-blk /dev/nvme1n1,no_bst_block` Note: When io_uring is enabled, the BST_BLOCK logic would be bypassed. Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]>

ACRN_IOEVENTFD_FLAG_ASYNCIO is not set when unregister ioeventfd in the current implementation which will cause the old asyncio_desc will be remained in hypervisor link list when switching from OVMF to kernel. Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

ACRN virtio devices are using a per device mutex to protect the concurrent operations on the device's PIO/MMIO. This introduces big contention in fast IO hence downgrades the IO performance, for example virtio-blk with asyncio enabled. This patch introduces per queue mutex to relieve such issues. Currently the per queue mutex is only used in the asycio path when iothread is enabled. Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

Virtio legacy device (ver < 1.0) uses a single PIO for all virtqueues. Notifications from different virtqueues are implemented by writing virtqueue index to the PIO. Writing different values to the same addr needs to be mapped to different eventfds by asyncio. This is called data match feature of asyncio. v3 -> v4: * Update the definition of `struct asyncio_desc` Use `struct acrn_asyncio_info` inside it, instaed of defining the duplicated fileds. * Update `add_asyncio` to use `memcpy_s` rather than assigning all the fields using 5 assignment statements. * Update `asyncio_is_conflict` for coding style 120-character line is sufficient to write all conditions. * Update the checks related to `wildcard` Because we require every conditional clause to have a Boolean type in the coding guideline. v2 -> v3: No change v1 -> v2: No change Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]> Signed-off-by: Shiqing Gao <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

Virtio-blk can support multiple virtqueues (mq) which is negotiated between FE and BE by the feature bit VIRTIO_BLK_F_MQ. The virtqueue number of virtio-blk can be specified by "mq=x" in the parameter. For example: "virtio-blk,iothread,mq=2,..." Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

block_if is the backend of ahci and virtio-blk. Only one queue is supported by block_if now. Several worker threads are created as the thread pool for the queue. One BIG mutex is used for the queue and thread operation. With this patch block_if can support multiple queues and each queue is backed by several worker threads. blockif_req can be submited/enqueued into one specified queue. By spliting into several queues contention from the BIG mutex can be relieved/eliminated. This is used to support virtio-blk multiple queues feature. Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

io_uring is a high-performance asynchronous I/O framework, primarily designed to improve the efficiency of input and output (I/O) operations in user-space applications. This patch enables io_uring in block_if module. It utilizes the interfaces provided by the user-space library `liburing` to interact with io_uring in kernel-space. To build the acrn-dm with io_uring support, `liburing-dev` package needs to be installed. For example, it can be installed like below in Ubuntu 22.04. sudo apt install liburing-dev In order to support both the thread pool mechanism and the io_uring mechanism, an acrn-dm option `aio` is introduced. By default, thread pool mechanism is selected. - Example to use io_uring: `add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback,aio=io_uring` - Example to use thread pool: `add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback,aio=threads` - Example to use thread pool (by default): `add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback` v2 -> v3: * Update iothread_handler - Use the unified eventfd interfaces to read the counter value of the ioeventfd. - Remove the while loop to read the ioeventfd. It is not necessary because one read would reset the counter value to 0. * Update iou_submit_sqe to return an error code The caller of iou_submit_sqe shall check the return value. If there is NO available submission queue entry in the submission queue, need to break the while loop. Request can only be submitted when SQE is available. v1 -> v2: * move the logic of reading out ioeventfd from iothread.c to virtio.c, because it is specific to the virtqueue handling. Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

Prior to this patch, one single iothread instance is created and initialized in the `main` function. This single iothread monitors all the registered fds and handles all the corresponding requests. It leads to the limited flexibility of the iothread support. To improve the flexibility of the iothread support, this patch does: - add the support of multiple iothread instances. `iothread_create` is introduced to create a certain number of iothread instances. It shall be called at first by each virtual device owner (such as virtio-blk BE) on initialization phase. Then, `iothread_add` can be called to add the to be monitored fd to the specified iothread. - update virtio-blk BE to let the acrn-dm option `iothread` accept a number as the number of iothread instances to be created. If `iothread` is contained in the parameters, but the number is not specified, one iothread instance would be created by default. Examples to specify the number of iothread instances: 1. Create 2 iothread instances `add_virtual_device 9 virtio-blk iothread=2,mq=2,/dev/nvme1n1,writeback,aio=io_uring` 2. Create 1 iothread instances (by default) `add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback,aio=io_uring` - update virtio-blk BE to separate the request handling of different virtqueues to different iothreads. The request from one or more virtqueues can be handled in one iothread. The mapping between virtqueues and iothreads is based on round robin. v1 -> v2: * add a mutex to protect the free ioctx slot allocation Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

To improve the performance of the virtual device who utilizes iothread (such as virtio-blk), this patch sets iothread nice value to PRIO_MIN, so that it could get higher priority on scheduling. This patch does: - introduce `set_thread_priority` to set the priority of the current running thread. The priority could be any value in the range PRIO_MIN to PRIO_MAX. Lower numerical value causes more favorable scheduling. - set iothread nice value to PRIO_MIN. Tracked-On: projectacrn#8612 Signed-off-by: Jian Jun Chen <[email protected]> Signed-off-by: Shiqing Gao <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

This patch adds an acrn-dm option `nocache` to bypass the Service VM's page cache. - By default, the Service VM's page cache is utilized. - If `nocache` is specified in acrn-dm parameters, the Service VM's page cache would be bypassed (opening the file/block with O_DIRECT flag). Example to bypass the Service VM's page cache: `add_virtual_device 5 virtio-blk iothread,mq=2,/dev/nvme2n1,writeback,nocache` Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

Use of O_DIRECT flag could be a performance option. But this flag may impose alignment restrictions on the length and address of user-space buffers and the file offset of I/Os. To support the use of O_DIRECT flag in block_if, this patch adds the support to handle the misaligned request. - When O_DIRECT flag is used (`nocache` is specified in acrn-dm parameters), * if the original I/O request is aligned, the original I/O request is submitted directly. * if the original I/O request is not aligned (either due to the buffer address/length misalignment, or the offset misalignment), the misaligned request is converted to an aligned request before submission. - When O_DIRECT flag is not used, the original I/O request is submitted directly. v1 -> v2: * cleanup the free() logic in `blockif_init_bounced_write` Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

This patch renames the iothread for better readability. For instance, the new name of the iothread for virtio-blk device looks like `iothr-0-blk9:0`. It could be helpful when tuning the performance and the CPU utilization. v1 -> v2: * add `const` qualifier for the input parameter of `iothread_create` Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

This patch updates the `iothread` option to specify the CPU affinity of the iothread. Setting the iothread's CPU affinity could benefit the Service VM's CPU utilization when Service VM owns limited dedicated CPUs. It could be helpful to ensure the I/O mediator Quality of Service (QoS). Once the performance tuning is done, the specific CPU affinity config could pass to acrn-dm directly, letting the deployment more easily. The format looks like below: iothread=<num_iothread>@<cpu_affinity> "@" is used to separate the following two settings: - the number of iothread instances - the CPU affinity settings for each iothread instance. The format of `cpu_affinity` looks like below: <cpu_affinity_0>/<cpu_affinity_1>/<cpu_affinity_2>/... 1. "/" is used to separate the CPU affinity setting for each iothread instance (sequentially). 2. char '*' can be used to skip the setting for the specific iothread instance. 3. the number of cpu_affinity_x vs. the number of iothread instances - If # of cpu_affinity_x is less than # of iothread instances, no CPU affinity settings for the last few iothread instances. - If # of cpu_affinity_x is more than # of iothread instances, the extra cpu_affinity_x are discarded. 4. ":" is used to separate different CPU cores for each CPU affinity setting. Examples to specify the CPU affinity of the iothread: 1. iothread=3@0:1:2/0:1 `add_virtual_device 9 virtio-blk iothread=3@0:1:2/0:1,mq=3,/dev/nvme1n1` a) 3 iothread instances are created. b) CPU affinity of iothread instances for this virtio-blk device: - 1st iothread instance <-> pins to Service VM CPU 0,1,2 - 2nd iothread instance <-> pins to Service VM CPU 0,1 - 3rd iothread instance <-> No CPU affinity settings 2. iothread=3@0/*/1 `add_virtual_device 9 virtio-blk iothread=3@0/*/1,mq=3,/dev/nvme1n1` a) 3 iothread instances are created. b) CPU affinity of iothread instances for this virtio-blk device: - 1st iothread instance <-> pins to Service VM CPU 0 - 2nd iothread instance <-> No CPU affinity settings - 3rd iothread instance <-> pins to Service VM CPU 1 v1 -> v2: * encapsulate one API in iothread.c to parse the iothread options, so that other BE can also use it. v2 -> v3: * introduce one API iothread_free_options to free the elements that are allocated dynamically in iothread_parse_options(). Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

When `io_uring` is used, `blockif_flush_cache` is missing when an WRITE operation is completed. `blockif_flush_cache` would flush the modified in-core data to the disk device according to the setting of the cache mode. Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

When multiple virtio-blk instances are created for one VM, using the same `static struct virtio_ops virtio_blk_ops` for all instances is buggy. It only works when all instances are created with the same number of the virtqueues. This patch fixes this issue by introducing a member in `struct virtio_blk` to store the ops info for each virtio-blk instance. Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]> Acked-by: Wang, Yu1 <[email protected]>

With current implementation, in blockif_dequeue/blockif_complete, if the current request is consecutive to any request in penq or busyq, current request's status is set to BST_BLOCK. Then, this request is blocked until the prior request, which blocks it, is completed. It indicates that consecutive requests are executed sequentially. This patch adds a flag `no_bst_block` to bypass such logic because: 1. the benefit of this logic is not noticeable; 2. there is a chance that a request is enqueued in block_if_queue but not dequeued when this logic is triggered along with the io_uring mechanism; Example to use this flag: `add_virtual_device 5 virtio-blk /dev/nvme1n1,no_bst_block` Note: When io_uring is enabled, the BST_BLOCK logic would be bypassed. Tracked-On: projectacrn#8612 Signed-off-by: Shiqing Gao <[email protected]> Acked-by: Wang, Yu1 <[email protected]>