[Documentation] Can CPUEP infer multithread with one session?

[kq566]

Describe the documentation issue

I have one Ort::Session. Can I use multithreads with one session? My code is roughly set up like this and works, but during execution, it doesn't utilize all 16 CPU cores.

Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(4);
Ort::Session session;

std::vector<std::thread> threads;
for (int i =0; i< 4; ++i)
{
    threads.push_back(session.Run());
}
for (auto t : threads)
{
    t.jion();
}

I would like to ask if this multi-threading approach is recommended. Is there a better way to implement multi-threaded inference to increase throughput?

Page / URL

No response

[skottmckay]

Sessions are stateless so you can call Run concurrently.

[kq566]

Sessions are stateless so you can call Run concurrently.

Thank you very much for your reply. I have another question. Since Run can be called concurrently, does each Run in my example use 4 CPU cores? Is it possible to fully utilize all 16 CPU cores?

[wejoncy]

Try setting kOrtSessionOptionsConfigAllowIntraOpSpinning

onnxruntime/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h

Line 118 in c5276ac

static const char* const kOrtSessionOptionsConfigAllowIntraOpSpinning = "session.intra_op.allow_spinning";

to 0, which might help

[kq566]

kOrtSessionOptionsConfigAllowIntraOpSpinning

@wejoncy Thank you very much for your reply. I tried this parameter, but it doesn't seem to work. I also noticed that when I set n threads, the total number of cores is n, and each thread seems to be limited to using only one CPU core? Are there any other settings that allow configuring the number of CPU cores each thread can use?

[wejoncy]

Actually, "use 4 CPU cores" in one session is expected in one session even you run this session concurrently. SetIntraOpNumThreads is the only one you can configure thread-nums.

you can set thread-numbers to 16 and run them in a batch. The session will use all cores then.