[Question]: Can using DMA-BUF in IBGDA improve performance?

[kwu130]

Question

When I enable DMA-BUF in IBGDA transport mode (it is enabled by default), my data buffer can be successfully registered using ibv_reg_dmabuf_mr(). However, registration of the control buffer using mlx5dv_devx_umem_reg_ex() fails with errno = 14 (Bad address). As a result, it falls back to registering the control buffer with mlx5dv_devx_umem_reg() instead.

In this scenario, the data buffer is registered via DMA-BUF, while the control buffer is registered through the non-DMA-BUF path.

My questions are:

Can this mixed configuration (DMA-BUF for the data buffer and non-DMA-BUF for the control buffer) provide any performance benefit?
Based on my testing, I did not observe any noticeable performance improvement.

Would registering both buffers via DMA-BUF potentially improve performance?

What could be the possible reasons for the failure of mlx5dv_devx_umem_reg_ex() with errno = 14 (Bad address)?

For reference, my environment is:

GPU: NVIDIA L20

Driver Version: 590.44.01

CUDA Version: 13.1

Linux kernel version: 4.18.0-553.54.1.el8_10.x86_64