Multithreaded Inference on GPU with only one session in Onnx Runtime

[BenBoguslavsky18]

Is there any way to enable multithreaded processing using the GPU while only having one session instance? I have tried SetIntraOpNumThreads, SetInterOpNumThreads, etc., but it has not shown any improvements in performance. My goal is to only have one instance of an object which contains the session, but be able to run inferencing on four threads.

Here is my code for setting up a session:

env = Ort::Env(OrtLoggingLevel::ORT_LOGGING_LEVEL_WARNING, "ONNX_DETECTION");
sessionOptions = Ort::SessionOptions();
sessionOptions.SetIntraOpNumThreads(6);
sessionOptions.SetInterOpNumThreads(1);
sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);


std::vector<std::string> availableProviders = Ort::GetAvailableProviders();
auto cudaAvailable = std::find(availableProviders.begin(), availableProviders.end(), "CUDAExecutionProvider");


// CUDA EP option
OrtCUDAProviderOptions cudaOption;

for (const auto& p : availableProviders) {
    std::cout << "Available EP: " << p << std::endl;
}

if (isGPU && (cudaAvailable == availableProviders.end()))
{
    std::cout << "GPU is not supported by your ONNXRuntime build. Fallback to CPU." << std::endl;
    std::cout << "Inference device: CPU" << std::endl;
}
else if (isGPU && (cudaAvailable != availableProviders.end()))
{
    try {
        std::cout << "Inference device: GPU" << std::endl;
        sessionOptions.AppendExecutionProvider_TensorRT_V2(*tensorrt_options);
        sessionOptions.AppendExecutionProvider_CUDA(cudaOption);
    }
    catch (const Ort::Exception& e) {
        std::cerr << "ORT Exception: " << e.what() << std::endl;
    }
}
else
{
    std::cout << "Inference device: CPU" << std::endl;
}

#ifdef _WIN32
std::wstring w_modelPath = utils::charToWstring(modelPath.c_str());
session = Ort::Session(env, w_modelPath.c_str(), sessionOptions);
#else
session = Ort::Session(env, modelPath.c_str(), sessionOptions);
#endif

Ort::AllocatorWithDefaultOptions allocator;

Ort::TypeInfo inputTypeInfo = session.GetInputTypeInfo(0);
std::vector<int64_t> inputTensorShape = inputTypeInfo.GetTensorTypeAndShapeInfo().GetShape();
this->isDynamicInputShape = false;

// checking if width and height are dynamic
if (inputTensorShape[2] == -1 && inputTensorShape[3] == -1)
{
    std::cout << "Dynamic input shape" << std::endl;
    this->isDynamicInputShape = true;
}

for (auto shape : inputTensorShape)
    std::cout << "Input shape: " << shape << std::endl;

inputNames.push_back("images");
outputNames.push_back("output0");

this->inputImageShape = cv::Size2f(inputSize);

---

And here is my multithreading setup:

// set thresholds
const float confThreshold = 0.5f;
const float iouThreshold = 0.3f;

// set model (YOLOv8) and image path
const std::string imagePath1 = "test4WaterMeter//testImage.jpg";
const std::string imagePath2 = "test4WaterMeter//testImage2.jpg";
const std::string modelPath = "test4WaterMeter//Yolov8//bestWaterY8.onnx";

// create images objects
cv::Mat image1 = cv::imread(imagePath1);
cv::Mat image2 = cv::imread(imagePath2);

std::vector<Detection> results;
YOLODetector detector(modelPath, 1, cv::Size(640, 640)); //this constructs the code above

std::vector<std::thread> threads;

for (int i = 0; i < 4; ++i) {
    threads.emplace_back([&detector, confThreshold, iouThreshold, i, &image1, &image2, &results, &inference_times]() {

        auto start = std::chrono::high_resolution_clock::now();

        for (int j = 0; j < 100; ++j) { // Each thread runs 100 inferences
            if (j % 2 == 0) {
                results = detector.detect(image1, confThreshold, iouThreshold);
            }
            else {
                results = detector.detect(image2, confThreshold, iouThreshold);
            }
        }

        auto end = std::chrono::high_resolution_clock::now();
        inference_times[i] = std::chrono::duration_cast<std::chrono::microseconds>(end - start);
        });
}

for (auto& t : threads) {
    if (t.joinable()) {
        t.join();
    }
}

---

Thanks.

[idruker-cerence]

A few things:

1. setting IntraOpNumThreads and InterOpNumThreads does make an improvement in the inference speed but probably not in the manner you expect. It does not help to run multiple tasks on the same GPU, but rather optimizes running one graph within one session. The speed-up depends on many factors including the number of GPU cores. There is an optimal number of the intra- and inter-op-num-threads, and if you exceed it, the speed stops increasing.
2. As far as I know, you can run multiple inferences in parallel in one session
3. Regarding your "And here is my multithreading setup". This code for assigning parallel tasks to different cores only works with CPU. It does not work for GPU. For GPU you should try the routines from C++17 standard library that get ExecutionPolicy as an argument, for example, for_each(std::execution::par_unseq, …). The standard library does not implement the parallelization mechanism but only provides standard API, therefore, you will have to compile your project with the TBB library that implements this mechanism