CUBLAS_STATUS_ALLOC_FAILED

[danrca]

Hi, I am new to onnx and I am trying to build the libraries in a docker container, but during the self tests I am noticing initialization errors and some of my tests are failing. Here is the messages that I keep seeing on the failed tests.

1: 2024-01-19 07:29:22.017350225 [E:onnxruntime:QAttention, inference_session.cc:1587 operator()] Exception during initialization: /onnxruntime/onnxruntime/core/providers/cuda/cuda_call.cc:122 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cublasStatus_t; bool THRW = true] /onnxruntime/onnxruntime/core/providers/cuda/cuda_call.cc:116 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cublasStatus_t; bool THRW = true] CUBLAS failure 3: CUBLAS_STATUS_ALLOC_FAILED ; GPU=0 ; hostname=rekor-desktop ; expr=cublasCreate(&cublas_handle_);

ONNX version 1.11
Linux: Ubuntu 20.04
Cuda 11.4
libcudnn8 8.2.4.15-1+cuda11.4
libcublas-11.4

Any help/guidance would be appreciated.

thanks

Dan