UCT/CUDA_IPC: Use buffer id to detect VA recylcing #10405
Conversation
| arg1 = (const void *)&key->ph; | ||
| arg2 = (const void *)®ion->key.ph; | ||
| #endif | ||
| if (memcmp(arg1, arg2, cmp_size) == 0) { |
There was a problem hiding this comment.
Does it makes sense to update the test to cover this behavior?
| #if HAVE_CUDA_FABRIC | ||
| cmp_size = sizeof(key->ph.buffer_id); | ||
| arg1 = (const void *)&key->ph.buffer_id; | ||
| arg2 = (const void *)®ion->key.ph.buffer_id; | ||
| #else | ||
| cmp_size = sizeof(key->ph); | ||
| arg1 = (const void *)&key->ph; | ||
| arg2 = (const void *)®ion->key.ph; | ||
| #endif |
There was a problem hiding this comment.
- need to fix indent
- any reason to not compare buffer_id also for legacy, regardless of HAVE_CUDA_FABRIC?
There was a problem hiding this comment.
any reason to not compare buffer_id also for legacy, regardless of HAVE_CUDA_FABRIC?
I had this to begin with but key->ph is of type CUipcMemHandle when HAVE_CUDA_FABRIC is not declared. So we would need to change the type if we want to also include buffer_id.
This PR solves the case where cuda reused pointer value (alloc/free/alloc), and failed to detect that it's actually not same allocation instance, right? |
| @@ -124,7 +124,7 @@ static ucs_status_t uct_cuda_ipc_close_memhandle(uct_cuda_ipc_cache_region_t *re | |||
| (CUdeviceptr)region->mapped_addr, region->key.b_len)); | |||
| } | |||
| } else if (region->key.ph.handle_type == UCT_CUDA_IPC_KEY_HANDLE_TYPE_MEMPOOL) { | |||
| return UCT_CUDADRV_FUNC_LOG_WARN(cuMemPoolDestroy(region->key.ph.pool)); | |||
| return UCT_CUDADRV_FUNC_LOG_WARN(cuMemFree((CUdeviceptr)region->mapped_addr)); | |||
There was a problem hiding this comment.
i did not understand why this change? previously we were erroneously freeing mem pool?
There was a problem hiding this comment.
@tvegas1 It not erroneous but excessive. If a suballocation from the same mempool gets sent over in the future, then we would have benefited from using the same imported mempool but closememhandle occurs when there is VA recycling detected. This change addresses the problem by deferring all imported mempool releases to the point of UCP context destruction. The drawback of this approach as with previous IPC handling is that if the exporter explicitly destroys the mempool, there is a reference on the importer side until context destruction. This needs to be separately handled for legacy memory and newer allocations like VMM and Mallocasync in a different PR.
Why/What?
Export handles are used for VA recycling check today but this is not guaranteed to be unique for newer memory allocators like VMM, and CUDA memory pools. We need to switch to using buffer_id attribute to perform this check.