{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":26539525,"defaultBranch":"master","name":"Savu","ownerLogin":"DiamondLightSource","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2014-11-12T14:24:19.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/1233618?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1697449979.0","currentOid":""},"activityList":{"items":[{"before":null,"after":"a9fe7c9e3994c81116d8df2c9a10d6c85579836c","ref":"refs/heads/fix-image-stitching","pushedAt":"2023-10-16T09:52:59.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"yousefmoazzam","name":"Yousef Moazzam","path":"/yousefmoazzam","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13002919?s=80&v=4"},"commit":{"message":"Fix `ImageStitching` plugin hanging in multi-node MPI runs\n\nIn multi-node MPI runs via `savu_launcher.sh` (and possible other cases\ninvolving MPI) which included the `ImageStitching` plugin, it was\nobserved that the `ImageStitching` plugin never completed its processing\nand instead was \"hanging\" indefinitely.\n\nThe reason for this was found to be related to the calculation of\nstatistics. There was a check for 4D data in the calculation of\nstatistics, and if the data was labelled by Savu as 4D, to then perform\nsome extra statistics calculations on the 4D data assuming that the 4D\ndata contained volume data (most likely, assuming that the 4D data was a\ngroup of multiple 3D volumes).\n\nSavu labels the output of `LfovLoader` as being 4D, to mean either \"two\nstacks of projections to stitch together\" or \"two stacks of sinograms to\nstitch together\". `ImageStitching` takes this \"4D\" output from\n`LfovLoader` and then converts it to 3D.\n\nIn the `Statistics` class, the `setup_4d()` method applies some logic to\ndetermine if the plugin outputs 4D slices or not. The logic as it is\ncurrently concludes that `ImageStitching` outputs 4D slices, which is\nincorrect. Statistic calculations based on the assumption that the\nslices are volume data were then attempted to be applied to non-volume\ndata.\n\nSomewhere in the volume statistics calculations, at least one of the MPI\nprocesses \"hangs\", and the other MPI processes which encountered no\nproblem (for some unknown reason) are left waiting indefinitely at the\nMPI barrier in `PluginDriver._run_plugin_instances()`.\n\nRegarding the \"fix\" for this: the reasoning behind why the logic for\ndetermining if the output is 4D slices is set up this particular way is\nnot known. So, there are some reservations in \"correcting\" this for the\ncase of `ImageStitching`, out of concern for potentially breaking other\ncases where 4D slices are outputted/relevant.\n\nTherefore, the strategy to fix this situation is to instead:\n- query the pattern(s) of the input dataset(s)\n- check if any of them are the patterns that `ImageStitching` works on,\n  which are considered 4D by Savu but are *not* volumes\n- if any of the patterns of the input datasets match the patterns that\n  are connected to stitching images, do *not* perform volume statistics\n  calculations on the output slices","shortMessageHtmlLink":"Fix <code>ImageStitching</code> plugin hanging in multi-node MPI runs"}},{"before":"6a43cf74a63318d6e498b0c8d40d202edcc62f22","after":"6b1fe6309275855ef21d1ae7b7ea26427b57f330","ref":"refs/heads/slurm-migration","pushedAt":"2023-10-16T08:41:59.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"yousefmoazzam","name":"Yousef Moazzam","path":"/yousefmoazzam","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13002919?s=80&v=4"},"commit":{"message":"Add SLURM batch script for single CPU/GPU job\n\nThe flag `-u` can be passed to `savu_launcher.sh` to run a single\nCPU/GPU cluster job with SLURM.","shortMessageHtmlLink":"Add SLURM batch script for single CPU/GPU job"}},{"before":"ce838b850ad24fd19969ce2147cbf721bd3b57ba","after":"6a43cf74a63318d6e498b0c8d40d202edcc62f22","ref":"refs/heads/slurm-migration","pushedAt":"2023-07-11T16:07:32.000Z","pushType":"push","commitsCount":5,"pusher":{"login":"yousefmoazzam","name":"Yousef Moazzam","path":"/yousefmoazzam","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13002919?s=80&v=4"},"commit":{"message":"Add SLURM batch script for single CPU/GPU job\n\nThe flag `-u` can be passed to `savu_launcher.sh` to run a single\nCPU/GPU cluster job with SLURM.","shortMessageHtmlLink":"Add SLURM batch script for single CPU/GPU job"}},{"before":"8c81ea4c2fb207aefd83ab595148c8ef985782c3","after":"ce838b850ad24fd19969ce2147cbf721bd3b57ba","ref":"refs/heads/slurm-migration","pushedAt":"2023-03-16T09:09:18.844Z","pushType":"push","commitsCount":1,"pusher":{"login":"yousefmoazzam","name":"Yousef Moazzam","path":"/yousefmoazzam","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13002919?s=80&v=4"},"commit":{"message":"Remove copy of process list file to log folder in `savu_launcher.sh`\n\nThe way that this SLURM-migrated version of `savu_launcher.sh` is\nintended to be run is by first remotely logging onto `wilson`, and then\nrunning the launcher script via the `savu_mpi` alias. Therefore, all\ncommands in `savu_launcher.sh` are run on `wilson`.\n\nIn particular, there is a `cp` command that is run in\n`savu_launcher.sh`, seemingly wanting to copy the original prcoess list\nfile to the log folder associated with the Savu job. However, it is\npossible that there exist directories on compute nodes which are not\navailable on `wilson` in which a process list file may reside (for\nexample, currently `/mnt/gpfs03` is available on Hamilton nodes, but not\non `wilson`).\n\nIf a Savu job is submitted when the process list file is \"visible\" on\ncompute nodes but not on `wilson`, this causes the `cp` command to fail,\nand eventually cascades down to cause the Savu run to fail.\n\nAfter some observations of what files are copied by `savu_launcher.sh`\nvs. by Savu itself, and some inspection of the Savu source code, it\nappears that Savu itself also copies the process list file to the log\nfolder (seemingly also taking into account if more than one MPI process\nexists, to make sure that only one process attempts to do the file-save\noperation?).\n\nTherefore, it appeared unnecessary for `savu_launcher.sh` to do the `cp`\nat all, and after removing it and checking if the process list file was\ncopied to the run folder or not, it was indeed still copied without\nissue.\n\nHence, this change has removed the copy of the process list file from\n`savu_launcher.sh`, while still preserving the same behaviour as before.","shortMessageHtmlLink":"Remove copy of process list file to log folder in <code>savu_launcher.sh</code>"}},{"before":"587aec0a423bca46809d9f8b22628030c6daadff","after":"8c81ea4c2fb207aefd83ab595148c8ef985782c3","ref":"refs/heads/slurm-migration","pushedAt":"2023-03-08T15:44:47.730Z","pushType":"push","commitsCount":1,"pusher":{"login":"yousefmoazzam","name":"Yousef Moazzam","path":"/yousefmoazzam","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13002919?s=80&v=4"},"commit":{"message":"Remove references to `infile` var related to old dev mode\n\nSince 6a5b3b07 removed the notion of \"dev mode\" being initiated by a\nfile of job parameter values, this `infile` variable is no longer\nnecessary.","shortMessageHtmlLink":"Remove references to <code>infile</code> var related to old dev mode"}},{"before":"b8c58aacd97f0717180e5991ec8242e32232162a","after":"587aec0a423bca46809d9f8b22628030c6daadff","ref":"refs/heads/slurm-migration","pushedAt":"2023-03-07T13:42:37.613Z","pushType":"push","commitsCount":4,"pusher":{"login":"yousefmoazzam","name":"Yousef Moazzam","path":"/yousefmoazzam","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/13002919?s=80&v=4"},"commit":{"message":"Remove unnecessary creation of temp file","shortMessageHtmlLink":"Remove unnecessary creation of temp file"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAADl-JpDQA","startCursor":null,"endCursor":null}},"title":"Activity · DiamondLightSource/Savu"}