MLFlow exception error while training segmentation model

[amitjc]

Dear Experts,

I have been training segmentation model for segmentation of liver, mass, portal vein and hepatic veins. While having trained the model on some 20 cases, so far, I recently encountered mlflow error and unable to train further.

Please find attached the complete log here.

Seek guidance to overcome the error.

Thank you.

Kind Regards,
Amit.

[diazandr3s]

Hi @amitjc,

Thanks for commenting about this and sorry for the late reply.

From the logs, it seems the train mean dice class doesn't have the correct arguments:

File "/root/.pyenv/versions/3.10.14/lib/python3.10/site-packages/mlflow/store/tracking/file_store.py", line 782, in _get_metric_from_line
raise MlflowException(
mlflow.exceptions.MlflowException: Metric 'train_mean_dice' is malformed; persisted metric data contained 1 fields. Expected 2 or 3 fields.

can I ask how did you modify this class for your custom model?

[amitjc]

Hello @diazandr3s,

Thanks for your response.

I didn't modify file_store.py.
Only configured the segmentation.py under /trainer/configs.

[diazandr3s]

Hi @amitjc,

I'm sorry I just replied to this.
This is strange. Could you please try again with a fresh MONAI Label installation using the latest release and let us know?

[amitjc]

Hello @diazandr3s ,

The latest installation doesn't throw this error, however, this will need to be retrained all over again on my data!