"Running FacePredictor: au" gets stuck with no further progress

[SasaniP]

I am running FaceAnalyzer continuously for a set of images. My purpose is to detect the faces and then recognise the detected faces against a set of reference facial embeddings. The first half of the FaceAnalyzer works fine. But at some point, FaceAnalyzer gets stuck when iterating through the images. I am working on Google Colab.

The following is the verbose output of correctly running FaceAnalyzer.

{"asctime": "2025-04-28 05:12:10,855", "levelname": "INFO", "message": "Initializing FaceAnalyzer"}
{"asctime": "2025-04-28 05:12:10,855", "levelname": "INFO", "message": "Initializing BaseReader"}
{"asctime": "2025-04-28 05:12:10,870", "levelname": "INFO", "message": "Initializing FaceDetector"}
25 166 12959 [[[0 0 0]
  [0 0 0]
  [0 0 0]
  ...
image saved at /tmp/temp_audio_directorypon9vgko/12959.png
{"asctime": "2025-04-28 05:12:11,129", "levelname": "INFO", "message": "Initializing FaceUnifier"}
{"asctime": "2025-04-28 05:12:11,142", "levelname": "INFO", "message": "Initializing FacePredictor objects"}
{"asctime": "2025-04-28 05:12:11,142", "levelname": "INFO", "message": "Initializing FacePredictor embed"}
{"asctime": "2025-04-28 05:12:11,346", "levelname": "INFO", "message": "Initializing FacePredictor verify"}
{"asctime": "2025-04-28 05:12:12,571", "levelname": "INFO", "message": "Initializing FacePredictor fer"}
{"asctime": "2025-04-28 05:12:12,850", "levelname": "INFO", "message": "Initializing FacePredictor au"}
{"asctime": "2025-04-28 05:12:13,503", "levelname": "INFO", "message": "Initializing FacePredictor va"}
{"asctime": "2025-04-28 05:12:13,549", "levelname": "INFO", "message": "Initializing FacePredictor deepfake"}
{"asctime": "2025-04-28 05:12:14,256", "levelname": "INFO", "message": "Initializing FacePredictor align"}
{"asctime": "2025-04-28 05:12:14,458", "levelname": "INFO", "message": "Initializing BaseUtilizer objects"}
{"asctime": "2025-04-28 05:12:14,459", "levelname": "INFO", "message": "Initializing BaseUtilizer align"}
{"asctime": "2025-04-28 05:12:14,480", "levelname": "INFO", "message": "Initializing BaseUtilizer draw_boxes"}
{"asctime": "2025-04-28 05:12:14,483", "levelname": "INFO", "message": "Initializing BaseUtilizer draw_landmarks"}
{"asctime": "2025-04-28 05:12:14,486", "levelname": "INFO", "message": "Running FaceAnalyzer"}
{"asctime": "2025-04-28 05:12:14,487", "levelname": "INFO", "message": "Reading image", "input": "/tmp/temp_audio_directorypon9vgko/12959.png"}
{"asctime": "2025-04-28 05:12:14,499", "levelname": "INFO", "message": "Detecting faces"}
{"asctime": "2025-04-28 05:12:14,681", "levelname": "INFO", "message": "Number of faces: 0"}

The following is the output when FaceAnalyzer gets stuck at au model.

{"asctime": "2025-04-28 05:12:14,946", "levelname": "INFO", "message": "Initializing FaceAnalyzer"}
{"asctime": "2025-04-28 05:12:14,947", "levelname": "INFO", "message": "Initializing BaseReader"}
{"asctime": "2025-04-28 05:12:14,960", "levelname": "INFO", "message": "Initializing FaceDetector"}
25 167 12999 [[[0 0 0]
  [0 0 0]
  [0 0 0]
  ...
image saved at /tmp/temp_audio_directorypon9vgko/12999.png
{"asctime": "2025-04-28 05:12:15,224", "levelname": "INFO", "message": "Initializing FaceUnifier"}
{"asctime": "2025-04-28 05:12:15,243", "levelname": "INFO", "message": "Initializing FacePredictor objects"}
{"asctime": "2025-04-28 05:12:15,243", "levelname": "INFO", "message": "Initializing FacePredictor embed"}
{"asctime": "2025-04-28 05:12:15,446", "levelname": "INFO", "message": "Initializing FacePredictor verify"}
{"asctime": "2025-04-28 05:12:16,667", "levelname": "INFO", "message": "Initializing FacePredictor fer"}
{"asctime": "2025-04-28 05:12:16,945", "levelname": "INFO", "message": "Initializing FacePredictor au"}
{"asctime": "2025-04-28 05:12:17,597", "levelname": "INFO", "message": "Initializing FacePredictor va"}
{"asctime": "2025-04-28 05:12:17,643", "levelname": "INFO", "message": "Initializing FacePredictor deepfake"}
{"asctime": "2025-04-28 05:12:18,340", "levelname": "INFO", "message": "Initializing FacePredictor align"}
{"asctime": "2025-04-28 05:12:18,539", "levelname": "INFO", "message": "Initializing BaseUtilizer objects"}
{"asctime": "2025-04-28 05:12:18,540", "levelname": "INFO", "message": "Initializing BaseUtilizer align"}
{"asctime": "2025-04-28 05:12:18,561", "levelname": "INFO", "message": "Initializing BaseUtilizer draw_boxes"}
{"asctime": "2025-04-28 05:12:18,564", "levelname": "INFO", "message": "Initializing BaseUtilizer draw_landmarks"}
{"asctime": "2025-04-28 05:12:18,566", "levelname": "INFO", "message": "Running FaceAnalyzer"}
{"asctime": "2025-04-28 05:12:18,567", "levelname": "INFO", "message": "Reading image", "input": "/tmp/temp_audio_directorypon9vgko/12999.png"}
{"asctime": "2025-04-28 05:12:18,585", "levelname": "INFO", "message": "Detecting faces"}
{"asctime": "2025-04-28 05:12:19,073", "levelname": "INFO", "message": "Number of faces: 13"}
{"asctime": "2025-04-28 05:12:19,073", "levelname": "INFO", "message": "Unifying faces"}
{"asctime": "2025-04-28 05:12:19,077", "levelname": "INFO", "message": "Predicting facial features"}
{"asctime": "2025-04-28 05:12:19,077", "levelname": "INFO", "message": "Running FacePredictor: embed"}
{"asctime": "2025-04-28 05:12:19,291", "levelname": "INFO", "message": "Running FacePredictor: verify"}
{"asctime": "2025-04-28 05:12:29,153", "levelname": "INFO", "message": "Running FacePredictor: fer"}
{"asctime": "2025-04-28 05:12:30,713", "levelname": "INFO", "message": "Running FacePredictor: au"}

After the "message": "Running FacePredictor: au", it stays at this point for hours and is not responsive until the session is terminated.
Installed versions.

Python 3.11.12

librosa==0.10.1
audioread==3.0.1
scikit-learn==1.2.2
pydub==0.25.1
moviepy==1.0.3
scenedetect[opencv-headless]==0.6.3
pyannote.core==5.0.0
"torch==1.13.1+cu117"
"torchvision==0.14.1+cu117"
"torchaudio==0.13.1"
numpy==1.25.2
facetorch==0.5.0

Any help and guidance would be appreciated.

[dongyunjinyu]

#56 This question has been raised before, and it still persists.
I'm experiencing the same issue on CUDA 12.1 and CUDA 11.8, but not on CUDA 11.3.

[dongyunjinyu]

@dongyunjinyu In fact, CUDA 11.3 also has a card phenomenon, but it is not stuck like other versions of CUDA, but only 100 to 3000 milliseconds, and it occurs randomly. In most cases, it will not be stuck.