Getting worse times for DirectML than CPU.

[elephantpanda]

I managed to get DirectML to work, but it is giving me worse times than just running on CPU.

I have Intel UHD integrated graphics. Is this expected? Or could I be doing something wrong?

[RyanUnderhill]

It's theoretically possible for DirectML to be slower with slow integrated graphics and a fast CPU. Especially if there's other load on the GPU.

Can you give more details, like what model of CPU, how much slower, and possibly verify the CPU/GPU usage while it runs using Task Manager/etc?

[elephantpanda]

It's theoretically possible for DirectML to be slower with slow integrated graphics and a fast CPU. Especially if there's other load on the GPU.

Can you give more details, like what model of CPU, how much slower, and possibly verify the CPU/GPU usage while it runs using Task Manager/etc?

Hi I bought an extra 8GB of RAM. I am seeing 2x speed up of running inference with DirectML compared to CPU. So that's good.
But the DirectML is creating a long time to load the sessions. So overall it might equal out.

P.S. I tried running onnxruntime on a separate thread and it seems to work fine.

[fdwr]

@pauldog Nice, 2x speed. So it's one-and-done execution then? It's helpful to hold the model session in memory if you're going to be reusing it multiple times.

[elephantpanda]

@pauldog Nice, 2x speed. So it's one-and-done execution then? It's helpful to hold the model session in memory if you're going to be reusing it multiple times.

Yes, I will be running the inference multiple times. It's nice to see the 2x speed increase even on Intel integrated GPU. I'm not sure why the initial setup takes so long though... seeing as the CPU and GPU are both using the same shared memory. But I don't know much about how these things work. It seems like the GPU has to go through a "virtual inference" session to set everything up even before it does it's first real inference. Which I guess it needs to do to calculate how much memory it would need. It would be nice if we could store that information in a file. It might make it quicker next time to start a GPU session. This is just a guess because I don't know how it works! e.g. store the state of the GPU memory, so that it could just be restored with a single memory transfer.

[elephantpanda]

Well I ended up just getting a Shadow PC, so now the DirectML is working very fast now.

My laptop has corei3 Intel UHD graphics. And also I have too little RAM to run any big onnx.

I think it was only about 50% slower than CPU.

[RyanUnderhill]

Interesting, looping in @fdwr as they might want to hear this.

[fdwr]

could I be doing something wrong?

@pauldog: Doubtful. We've seen a wide gamut of performance depending on:

• The GPU model (integrated vs discrete, old vs new...),
• Size of the model (small/large)
• Size of tensors (upload/download cost and time)
• Number of CPU cores vs GPU cores
• The current power plan (e.g. some laptops put GPU's into a lower power mode until the driver detects significant computational work after some duration)

So there are cases where multithreaded CPU execution with 0 extra data copy time (no CPU <-> GPU transition) returns results quicker than processing it on the GPU, like small incremental workloads with tiny batch sizes requiring low latency (e.g. background audio filtering). GPU wins with workloads of larger tensors, saturated operations, and repeated frame-over-frame execution, on newer GPU's, discrete GPU's, and high-power plans.