Running multiple parallel inferences on the same model using the CUDA provider.

[xgirones]

I am using a tiny model for video filtering. To achieve better performance, I launch parallel inferences for a group of video frames. Since the CUDA provider appears to serialize concurrent calls to the same session, I create a pool of sessions of the same model. So far, this approach is working fine, but my concern is that I am wasting GPU resources due to static session state replication. Is there a way to create clones of a session? I am aware of the new 'CreateSessionWithPrepackedWeightsContainer' API call, but I do not how to use it to share all the static data of a session. What is the most efficient way to run parallel inferences on the same model using the CUDA provider?

[snnn]

Since the CUDA provider appears to serialize concurrent calls to the same session

If that's true, we should fix this. Do you have more details? Do you know which part of code restricts the concurrency?

[xgirones]

Thanks for your answer. Today I compiled a recent version of the ORT and ran several tests aiming to find the bottleneck, and the issue seems to be gone. I apologize for the false alarm.

Months ago I was experiencing performance problems when running multiple parallel inferences on the same CUDA session. As a workaround, I created a pool of session objects for the same model, so my question was more about how I could get the different sessions to share static data. But this is no longer necessary because in the version I have tested today, one session per model is enough.

[pranavsharma]

Just fyi. CreateSessionWithPrepackedWeightsContainer has nothing to do with CUDA.

[xgirones]

Thanks for the clarification. I imagine it works for all providers, but I was wondering if I could use it to allow CUDA sessions to share static data.

[pranavsharma]

No, it doesn't work for all providers. It's limited to CPU only so far.