Help with training using Deepedit and 3D Slicer

[mlluo]

Hi everyone, I'm currently trying to train using MONAILabel deepedit (dynunet) in 3D slicer, using a subset of the autoPETII challenge images (290 images total). The goal is to train a lesion segmentation model using active learning. The issue I'm running into is that it appears the training keeps getting stuck on the first epoch after using ~8-10 images and will not advance further. Everything works smoothly with the first few images, and I'm training the model after each image is segmented. I'm using NVIDIA GeForce RTX 4080, driver version 552.41, CUDA version 12.4. Any guidance would be appreciated, thanks!

[diazandr3s]

Hi @mlluo,

I'm sorry for the late reply.

For completeness, I'm attaching the logs you've shared. app.log

From the logs, I see you're using 7 samples for training and 2 for validation:

[2024-10-25 14:49:52,237] [20496] [MainThread] [INFO] (monailabel.tasks.train.basic_train:414) - Total Records for Training: 7
[2024-10-25 14:49:52,237] [20496] [MainThread] [INFO] (monailabel.tasks.train.basic_train:415) - Total Records for Validation: 2

It seems the 7 training samples can't be fit in the GPU you're using for this. You might need a bigger GPU.

Can you please share the volume/image sizes? Also, could you please try changing the loader before training as shown in this video?
Training will be slower that you could keep using the same GPU.

dataset.mp4

Hope this helps,