Simulation of user clicks - robot user

[Zrrr1997]

Hi,

I was wondering where I can take a look at the code for simulating user clicks during training. I read through some papers to get to know different ways of simulating user clicks, ranging from just clicking on the objects' centers to DIOS's 3 Strategies and many others...

I saw that you mention in this arXiv paper that you follow the protocol of IFSeg/DeepGrow to simulate user clicks. I see that they use the Chamfer distance transform to induce probability values of each pixel to be sampled as a positive/negative click. I tried to find how this is implemented in MONAILabel for DeepGrow but only got to the Interaction module in the docs.

Is this in the right direction or should I dig deeper to see how this is implemented in practice?

Another, more general question, is how would be the best way to go when you want to implement your own sampling strategy, say following DIOS's method, although it is aimed at natural images. Where would be a good starting point in implementing such a custom method?

Best regards, and as always, keep up the good work!
Zdravko

[diazandr3s]

Hi @Zrrr1997,

Thanks for opening this discussion. MONAI and MONAI Label will definitely benefit a lot from this contribution.

I was wondering where I can take a look at the code for simulating user clicks during training.

There are two parts: the initial pre-transforms and Interaction transforms. Here is the list of pre-transforms that initially assign random clicks: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/deepedit.py#L114-L117

Then there is the Interaction class ... This is connected to your second question:

I saw that you mention in this arXiv paper that you follow the protocol of IFSeg/DeepGrow to simulate user clicks. I see that they use the Chamfer distance transform to induce probability values of each pixel to be sampled as a positive/negative click.

Once we prepare the input tensor with initial clicks, there is an Interaction class with transforms that simulate user clicks based on the discrepancy. Here are the click interaction transforms: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/deepedit.py#L82-L97

These 3 transforms compute the discrepancy between ground truth and prediction, compute the distance and then add the clicks to the input tensor.

Another, more general question, is how would be the best way to go when you want to implement your own sampling strategy, say following DIOS's method, although it is aimed at natural images. Where would be a good starting point in implementing such a custom method?

I'd suggest using MONAI transforms for that. I think MONAI/MONAI Label have what is needed for this work.

BTW, we've just started CVAT support. The idea is that you can use it as the UI for MONAI Label. Then you should be able to implement 2D segmentation methods as well.

Hope this helps

[Zrrr1997]

Thanks so much for getting back to me! The Chamfer distance transform I was looking for is implemented in scipy.ndimage.morphology.distance_transform_cdt and is used in the lines of code you have listed.

After looking at the code, I think it would be easy to add more robot users to MONAI Label by just implementing new transforms. So MONAI already provides an amazing codebase to develop your own simulated user clicks/scribbles.