RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE

[maxm224]

What

I am trying to get this to run on a NVIDIA T4. However I always get an error. When I run it on a cpu it works great. Just a bit too slow for larger files.

The error:

lh2dg 2022-11-13T10:55:35.504Z layer_outputs = layer( l
h2dg 2022-11-13T10:55:35.504Z encoder_outputs = self.encoder( 
lh2dg 2022-11-13T10:55:35.504Z File "/opt/conda/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 872, in forward
lh2dg 2022-11-13T10:55:35.504Z return forward_call(*input, **kwargs) 
lh2dg 2022-11-13T10:55:35.504Z return forward_call(*input, **kwargs) lh2dg 2022-11-13T10:55:35.505Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 
lh2dg 2022-11-13T10:55:35.505Z **RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)`**

[maxm224]

Also on CPU I noticed that for recordings longer than 1 minute, the program crashes. Is there something I can configure to make longer recordings work?