Adjust n_tiles to prevent OOM error

[postnubilaphoebus]

Hi, thank you for providing this great plugin for Napari!
Unfortunately, tensorflow crashed for me with a tif volume of dimensions 195 x 666 x 1024 and a size of 133MB:

2024-01-31 09:59:52.515119: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at conv_ops_3d.cc:167 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[1,32,195,666,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Could you add some input windows to adjust the argument n_tiles?
I suspect it may be able to help with OOM issues.

[maweigert]

Hi @postnubilaphoebus ,

It seems you ran model.predict_instances with the default n_tiles, which doesn't do any tiling at all. Try n_tiles=(1,2,4) (or something higher).

[postnubilaphoebus]

Hi @maweigert
Thank you for the quick reply.
I will try different tile arrangements. Am I correct to assume that tiling such that 64^3 images result is best because the model was trained on such data?
I.e. I would use input_scale = (0.985, 0.96, 1) and n_tiles = (3, 10, 16) for my case (195 x 666 x 1024).

[maweigert]

No, the size of tiles should not affect the result very much and thus should be as large as possible (ie. small values of n_tiles) such that it still fits into your GPU, which typically is on the order of 192^3 (thus n_tiles=(1,2,4) in your case).
Additionally, I would not scale the input as its almost isotropic anyway.