minor issue to clarify

[simmed00]

Great work for the detailed comparison. I notice a small issue, maybe critical to the comparison between nnUNet and other SAM-based methods. The dice calculation in the code (single_GPU_train_finetune_noprompt.py) includes background, but as far as I know, nnUNet by default does not include background for dice calculation. This may lead to overestimation of the performance in the SAM method, as background usually scores a dice close to 1. It would be better to double check. Would that affect the conclusion in the comparison with nnUNet or UNet based methods?

[haoyudong-97]

Thank you for pointing this issue out! The released code is slightly different from the one used during writing the paper. The released version reports the per-class dice and you can disable background class by conditions like "if cls == 0". The version used for the paper only computes the foreground dice.

[simmed00]

I want to follow-up on this dice metric issue. The code provided calculates dice=1 when no foreground pixel in both gt and in prediction. However, in nnUNet, this is calculated as np.Nan and excluded from the final mean calculation by np.nanmean. The method used in this code base will significantly overestimate average dice, especially for minor classes that do not appear in each slice. I urge for a more rigorous validation of the result

[haoyudong-97]

Thank you for brining this up. Dice has been criticized for the ambiguity when gt and pred are all 0s, and in this paper we avoid this problem by removing all empty slices, as described in Page 9: "In the pre-processing step of our datasets, we removed all data lacking target objects, specifically those images where the corresponding masks only consist of the background, for both training and testing purposes."

[simmed00]

Yes, all slice will have at least one foreground class. However, take for example a 3-class segmentation, on a particular slice/image, only class 0 and class 1 present (it is not an empty slice), class 2 both gt and pred are 0, in such case, this code base will give dice=1 for class 2, that's what I mean.