Thanks, all seem clear me. Pretty memory-hungry one actually.

One pretty simple way to reduce the amount of memory used is to make the weights array be float32 rather than float64; doing that would halve the amount of memory used by those 4 copies mentioned in the code comments.

I'll make an issue in the httomolibgpu repo so then we can address that at some point 🙂

One pretty simple way to reduce the amount of memory used is to make the weights array be float32 rather than float64; doing that would halve the amount of memory used by those 4 copies mentioned in the code comments.

I'll make an issue in the httomolibgpu repo so then we can address that at some point 🙂

Thanks, there is no reason to keep it 64bit.

FYI, I'm fixing the data type thing on the httomolibgpu side of things + adding a test. Wasn't there apparently. Then we can merge

Cool, thanks! When that's done I'll change stuff here to reflect the lower memory usage, and will check the memory hook test is still passing.

@yousefmoazzam memory hook test now pass (couldn't run 2160-80 as OOM on this machine), feel free to add/edit anything and merge with this one?

Ok that's fine, the 2160-80 passes on a wilson node, which is good.

I'll also run the 360 pipeline involving a recon method (gridrec I think?) which originally brought us to this problem, to verify it all works with that small change in the context of a proper run, and then will merge both PR's if all looks good.

Annoyingly, I'm getting a segfault in the reslice for some reason when running the pipeline bench_recon_gridrec_cpu360.yaml with 4 MPI processes, so I can't verify if it works in the context of a 4 process run.

With 2 MPI processes things are all fine (no segfault in reslice, and the sino_360_to_180 max slices estimaiton must be working ok since there's no CUDA OOM error).

I feel like that's sufficient evidence that this is working fine (though further investigation into why segfault in reslice is happening once more for 360 data should be done...), what do you think @dkazanc?

yes, certainly. We need to know why it is happening. Does it also happen for the GPU pipeline with 360 to 180 conversion or just the CPU one?

Adding this to be aware of:
When using gridrec the segfault or a "stall" can happen because gridrec relies on FFTs and it is RAM-hungry method. The GPU function sino_360_to_180 will define the number of slices to process and pass them to gridrec. This can result in a memory overflow on P100 nodes. It does work on the nodes where more memory available.
It is preferable to use FBP GPU method where possible.