Multiple input volumes may need resampling

[lassoan]

Please upload sample images (one pair of corresponding T2 and and ADC images in separate nrrd files) into a new "TestData" release. I'll later add code to register them into the Sample Data module. This will make it easy for users to test the model.

I've tried to run this on one of the ProstateX data sets from TCIA (I've uploaded them for easy access here: T2, adc). These images are of course not of the same size. I got this error: RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 384 but got size 84 for tensor number 1 in the list.

Full logs:

Processing started
Writing input file to C:/Users/andra/AppData/Local/Temp/Slicer/__SlicerTemp__2024-03-15_21+17+58.808/input-volume0.nrrd
Writing input file to C:/Users/andra/AppData/Local/Temp/Slicer/__SlicerTemp__2024-03-15_21+17+58.808/input-volume1.nrrd
Creating segmentations with MONAIAuto3DSeg AI...
Auto3DSeg command: ['C:/Users/andra/AppData/Local/slicer.org/Slicer 5.6.1/bin/../bin\\PythonSlicer.EXE', 'C:/D/SlicerMONAIAuto3DSeg/MONAIAuto3DSeg\\Scripts\\auto3dseg_segresnet_inference.py', '--model-file', 'C:\\Users\\andra\\.MONAIAuto3DSeg\\models\\prostate-v1.0.0\\model.pt', '--image-file', 'C:/Users/andra/AppData/Local/Temp/Slicer/__SlicerTemp__2024-03-15_21+17+58.808/input-volume0.nrrd', '--result-file', 'C:/Users/andra/AppData/Local/Temp/Slicer/__SlicerTemp__2024-03-15_21+17+58.808/output-segmentation.nrrd', '--image-file-2', 'C:/Users/andra/AppData/Local/Temp/Slicer/__SlicerTemp__2024-03-15_21+17+58.808/input-volume1.nrrd']
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
Model epoch 1234 metric 0.9796589612960815
Using crop_foreground
Using resample with  resample_resolution [0.625, 0.625, 3.5999999046325684]
Traceback (most recent call last):
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\monai\transforms\transform.py", line 140, in apply_transform
    return [_apply_transform(transform, item, unpack_items, lazy, overrides, log_stats) for item in data]
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\monai\transforms\transform.py", line 140, in <listcomp>
    return [_apply_transform(transform, item, unpack_items, lazy, overrides, log_stats) for item in data]
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\monai\transforms\transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\monai\transforms\utility\dictionary.py", line 971, in __call__
    d[self.name] = torch.cat(output, dim=self.dim)  # type: ignore
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\monai\data\meta_tensor.py", line 282, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\torch\_tensor.py", line 1418, in __torch_function__
    ret = func(*args, **kwargs)
RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 384 but got size 84 for tensor number 1 in the list.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\D\SlicerMONAIAuto3DSeg\MONAIAuto3DSeg\Scripts\auto3dseg_segresnet_inference.py", line 241, in <module>
    fire.Fire(main)
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\fire\core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\fire\core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\fire\core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\D\SlicerMONAIAuto3DSeg\MONAIAuto3DSeg\Scripts\auto3dseg_segresnet_inference.py", line 163, in main
    batch_data = inf_transform([image_files])
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\monai\transforms\compose.py", line 335, in __call__
    result = execute_compose(
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\monai\transforms\compose.py", line 111, in execute_compose
    data = apply_transform(
  File "C:\Users\andra\AppData\Local\slicer.org\Slicer 5.6.1\lib\Python\Lib\site-packages\monai\transforms\transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.utility.dictionary.ConcatItemsd object at 0x0000025FA646F460>
Processing failed with return code 1
Cleaning up temporary folder.
Processing failed after 19.30 seconds.

Processing finished.

If I resample either the t2 to have the same geometry as adc, or adc to have the same geometry as t2 then segmentation is successful.

Is this behavior expected?
Do we need to resample all the inputs to have the same geometry?

[diazandr3s]

Hi @lassoan,

Thanks for the update.
The input to the model is two tensors of the same size. We could either resize both volumes using a MONAI transform before running inference or request the user to resample them before running inference.
What do you think is the best way?

Please let me know your thoughts.

[lassoan]

If it can be automated then we have no reason to ask the user to do it. We can resample all the additional images to the geometry of the first image.

The only question is If the model should do it, the launcher script, or the Slicer module. The lower level the better (at lower level more information may be available about resampling requirements, optimal common resolution, etc) and it would be less dependencies on Slicer. However, if this is hard to implement at lower levels then it would be very easy to implement in the Slicer module.

[diazandr3s]

Yes, it makes sense to implement it with MONAI. Let me try to implement this as soon as the GTC is done :)

[diazandr3s]

Hi @lassoan,

Could you please try again with this PR? #33

It should resample all volumes to the same size as the first one.

[lassoan]

The error reported was fixed by #33.

There seems to be an error in the prostate segmentation processing workflow in that it only works if IJK axes areoriented in a certain way. Most likely the IJK to LPS matrix is simply ignored - it is not used for appropriately flipping/reordering IJK axes. However, this is a separate issue, so I'm closing this and adding a new issue for the image orientation bug: #35