zeMemFree slowdown in a loop

[colleeneb]

Point of Contact

Colleen

Contact Details

bertoni@anl.gov

Vendor/ALCF/other tickets/IDs

GSD-11962, NEO-17411

Reproducer Path

source/reproducers/l0/zememfree_slowdown/

Status

Open

Details

This code times a loop over calls to zeMemAllocDevice and zeMemFree where the frees are not in the same order as the mallocs. There is no memory leak, just the order of frees and mallocs are not the same. (This is similar to what a code does.)

Depending on the size of the allocs, the calls to zeFree slow down as iterations go on. for example:

# Total execution time: 124 microseconds for iteration 0
#    Time for just mallocs: 73 microseconds
#    Time for just frees: 51 microseconds
#
# and after 59500 iterations:
#
# Total execution time: 408 microseconds for iteration 59500
#    Time for just mallocs: 71 microseconds
#    Time for just frees: 337 microseconds

Is this expected? Should the higher-level language implement a memory pool? It looks like SYCL and OpenMP do this.

Intel says not expected.

This is relevant to an INCITE application (GENESIS application / OpenFanWing project)

Helpful environment variable (EnableDeviceUsmAllocationPool)

Trying to see if can backport newer runtime fixes to LTS2.

Is this a priority/blocking bug?

• Priority

ETA

being investigated, not sure if KMD or user-space, fixed being developed

[colleeneb]

Update: Setting

export NEOReadDebugKeys=1
export EnableDeviceUsmAllocationPool=1

will cause the reproducer to pass. In the full application, the time still increases, but more slowly.

[raffenet]

Is the code linked with MPICH? Does disabling the IPC handle cache affect the behavior at all?

export MPIR_CVAR_CH4_IPC_GPU_HANDLE_CACHE=disabled