How to use use_tf32 in Python CUDAExecutionProvider?

[DavorJordacevic]

Hello,

Can someone explain is it possible to use use_tf32 flag via Python API for CUDAExecutionProvider?

[mapluisch]

Hey Davor,

use_tf32 is enabled per default, according to the docs:

TensorFloat-32 is enabled by default.
Starting from ONNX Runtime 1.18, you can use this flag to disable it for an inference session.

Example python usage:

providers = [("CUDAExecutionProvider", {"use_tf32": 0})]
sess_options = ort.SessionOptions()
sess = ort.InferenceSession("my_model.onnx", sess_options=sess_options, providers=providers)

Hope that helps :)

[DavorJordacevic]

Hello,

Thank you. Is this version (1.18) available to download?

I am getting this error with the onnxruntime-gpu==1.17.1

*************** EP Error ***************
EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc:125 static onnxruntime::CUDAExecutionProviderInfo onnxruntime::CUDAExecutionProviderInfo::FromProviderOptions(const ProviderOptions&) [ONNXRuntimeError] : 1 : FAIL : provider_options_utils.h:148 Parse Unknown provider option: "use_tf32".
 when using [('CUDAExecutionProvider', {'use_tf32': 0})]
Falling back to ['CUDAExecutionProvider', 'CPUExecutionProvider'] and retrying.
****************************************

Here is the code:

providers = [("CUDAExecutionProvider", {"use_tf32": 0})]
sess_options = onnxruntime.SessionOptions()
ort_session = onnxruntime.InferenceSession('face_model.onnx', sess_options=sess_options, providers=providers)

[mapluisch]

Hi Davor,

ah sorry, I thought you wanted to enable it in the first place.

1.18 is not available yet, but given the quarterly release schedule it should probably be released soon?

Until then, you could set NVIDIA_TF32_OVERRIDE = 0, which should disable TF32.

[DavorJordacevic]

Great, thanks, this will also work for me.

[mapluisch]

You're welcome! Glad to hear :)