Merge pull request #380 from DiamondLightSource/param-sweep-runner

Add serial parameter sweep functionality

docs/source/howto/httomo_features.rst

@@ -6,6 +6,6 @@ HTTomo Features
 
    httomo_features/previewing
    httomo_features/centering
-   httomo_features/parameter_tuning
+   httomo_features/parameter_sweeping
 
 

docs/source/howto/httomo_features/centering.rst

@@ -93,7 +93,7 @@ Manual Centering
 =================
 
 Unfortunately, there could be various cases when :ref:`centering_auto` fails, e.g., the projection data is corrupted, incomplete, the object is outside the field of view of the detector, and possibly other issues.
-In that case, it is recommended to find the center of rotation manually. :ref:`parameter_tuning` can simplify such search significantly. 
+In that case, it is recommended to find the center of rotation manually. :ref:`parameter_sweeping` can simplify such search significantly.
 
 To enable manual centering, one would need to do the following steps:
 

docs/source/howto/httomo_features/parameter_sweeping.rst

@@ -0,0 +1,120 @@
+.. _parameter_sweeping:
+
+Parameter Sweeping
+^^^^^^^^^^^^^^^^^^
+
+What is it?
+===========
+
+Parameter sweeping refers to providing multiple values for a specific parameter
+of a method, and then running that method on its input data with the different
+values for that parameter.
+
+How would this be useful when processing data?
+==============================================
+
+This feature is typically used when prototyping a process list and it is
+difficult to guess a reasonable value of a method's parameter. There could be
+many reasons for this situation, such as being unfamiliar with the method, or
+working with unfamiliar data, etc.
+
+How are parameter sweeps defined in the process list YAML file?
+===============================================================
+
+There are two ways of specifying the values that a parameter sweep should be
+performed across:
+
+1. Specifying a range of values via start, stop and step values
+
+2. Manually specifying each value
+
+.. note:: A pipeline can only have 1 parameter sweep in it at a time. Any
+   pipelines with more than 1 parameter sweep defined in it will not be
+   executed, and an error message will be displayed.
+
+Specifying a range
+++++++++++++++++++
+
+The first way is done by providing a start, stop and step value. Along with this
+information, a special phrase :code:`!SweepRange` is used to "mark" in the YAML
+that the start, stop and step values are for defining a parameter sweep.
+
+The snippet below is defining a parameter sweep for the :code:`center` parameter
+of a reconstruction method, where the sweep starts at :code:`10`, ends at
+:code:`40` (similar to python slicing, the end value is not included), with steps
+of :code:`10` inbetween:
+
+.. code-block:: yaml
+
+    center: !SweepRange
+      start: 10
+      stop: 50
+      step: 10
+
+Specifying each value
++++++++++++++++++++++
+
+The second way is done by providing a list of values for a parameter, and again
+"marking" the list with a special phrase to denote that this list of values is
+defining a parameter sweep. The phrase in this case is :code:`!Sweep`.
+
+The snippet below is defining a parameter sweep for the :code:`size` parameter
+of a median filter method, where the sweep is across the two values :code:`3`
+and :code:`5`:
+
+.. code-block:: yaml
+
+    size: !Sweep
+      - 3
+      - 5
+
+Example
++++++++
+
+Below, :code:`!Sweep` is used in the context of a fully working minimal
+pipeline to sweep over the :code:`size` parameter of a median filter:
+
+.. literalinclude:: ../../../../tests/samples/pipeline_template_examples/testing/sweep_manual.yaml
+  :language: yaml
+
+How big should the input data be?
+=================================
+
+Due to the goal of parameter sweeps being to provide quick feedback to optimise
+a parameter value, it is typical to run a parameter sweep on a few sinograms,
+rather than the full data.
+
+As such, a parameter sweep run in HTTomo is constrained to run on data previewed
+to contain 7 sinogram slices or less. Meaning, in order to perform a parameter
+sweep in a pipeline, the input data must be cropped to 7 sinogram slices or less
+using the :code:`preview` parameter of the loader (see :ref:`previewing` for
+more details), otherwise the parameter sweep run will not execute and an error
+message will be displayed.
+
+What structure does the output data of a parameter sweep have?
+==============================================================
+
+When a parameter sweep is executed, the output of the method will be the set of
+middle slices from each individual result of the sweep (sinogram slices or recon
+slices), collected along the middle dimension.
+
+For example, suppose:
+
+- the input data is previewed to 3 sinogram slices and has shape
+  :code:`(1801, 3, 2560)`
+
+- a parameter sweep is performed on the :code:`center` parameter of a
+  reconstruction method in the pipeline, across 10 different CoR values
+
+In this case, each execution of the reconstruction method will produce 3 slices,
+as an array of shape :code:`(2560, 3, 2560)`. So, 10 arrays of shape
+:code:`(2560, 3, 2560)` will be produced.
+
+The middle slice from each array of 3 slices will be taken, resulting in 10
+reconstructed slices altogether. These 10 reconstructed slices will then be
+concatenated along the middle dimension and put into a separate array, resulting
+in the final data shape of :code:`(2560, 10, 2560)`.
+
+This output containing 10 slices will then be passed onto the next method in the
+pipeline; for example, a method to save the 10 slices as images for quick
+inspection.

docs/source/howto/httomo_features/previewing.rst

@@ -9,7 +9,7 @@ It also can be interpreted as a data cropping or data slicing operation.
 
 Reduction of the input data is often done to remove unnecessary/useless
 information, and to accelerate the processing time. It is also recommended to use
-when searching for optimal parameter values, see :ref:`parameter_tuning`. Skip to
+when searching for optimal parameter values, see :ref:`parameter_sweeping`. Skip to
 :ref:`previewing_enable` for information about how to use it in HTTomo.
 
 Previewing in the loader

docs/source/pipelines/yaml.rst

@@ -54,6 +54,13 @@ DLS Specific templates
 
 Parameter Sweeps templates
 ----------------------------
-Those templates demonstrate how to perform sweeps across multiple values a single parameter. See more on :ref:`parameter_tuning`.
 
-To be added in the forthcoming releases. 
+These templates demonstrate how to perform a sweep across multiple values of a
+single parameter (see :ref:`parameter_sweeping` for more details).
+
+.. dropdown:: Parameter sweep over 10 CoR values (`center` param) in recon
+   method, and saving the result as tiffs
+
+   .. literalinclude:: ../../../tests/samples/pipeline_template_examples/parameter-sweep-cor.yaml
+       :language: yaml
+       :emphasize-lines: 9-11,37-40

httomo/base_block.py

@@ -0,0 +1,124 @@
+from typing import Tuple
+
+import numpy as np
+
+from httomo.block_interfaces import BlockData, BlockTransfer, generic_array
+from httomo.runner.auxiliary_data import AuxiliaryData
+from httomo.utils import gpu_enabled, make_3d_shape_from_array, xp
+
+
+class BaseBlock(BlockData, BlockTransfer):
+    """
+    Base block class providing default implementations for the data
+    transferring/getting/setting behaviour needed for a block type to be processed by
+    implementors of `MethodWrapper`. Ie, this class provides default implementations for the
+    `BlockTransfer` and `BlockData` protocols.
+
+    Note that the data indexing behaviour described in `DataIndexing` is not implemented in
+    this class. If the default implementations for data transferring/getting/setting in this
+    class are acceptable: inherit from `BaseBlock`, override where necessary, and implement
+    `DataIndexing` in order to implement the `Block` protocol.
+    """
+
+    def __init__(self, data: np.ndarray, aux_data: AuxiliaryData) -> None:
+        self._data = data
+        self._aux_data = aux_data
+
+    def __dir__(self) -> list[str]:
+        """Return only those properties that are relevant for the data"""
+        return ["data", "angles", "angles_radians", "darks", "flats", "dark", "flat"]
+
+    @property
+    def data(self) -> generic_array:
+        return self._data
+
+    @data.setter
+    def data(self, new_data: generic_array):
+        self._data = new_data
+
+    @property
+    def aux_data(self) -> AuxiliaryData:
+        return self._aux_data
+
+    def _empty_aux_array(self):
+        empty_shape = list(self._data.shape)
+        return np.empty_like(self._data, shape=empty_shape)
+
+    @property
+    def angles(self) -> np.ndarray:
+        return self._aux_data.get_angles()
+
+    @angles.setter
+    def angles(self, new_angles: np.ndarray):
+        self._aux_data.set_angles(new_angles)
+
+    @property
+    def angles_radians(self) -> np.ndarray:
+        return self.angles
+
+    @angles_radians.setter
+    def angles_radians(self, new_angles: np.ndarray):
+        self.angles = new_angles
+
+    @property
+    def darks(self) -> generic_array:
+        darks = self._aux_data.get_darks(self.is_gpu)
+        if darks is None:
+            darks = self._empty_aux_array()
+        return darks
+
+    @darks.setter
+    def darks(self, darks: generic_array):
+        self._aux_data.set_darks(darks)
+
+    # alias
+    @property
+    def dark(self) -> generic_array:
+        return self.darks
+
+    @dark.setter
+    def dark(self, darks: generic_array):
+        self.darks = darks
+
+    @property
+    def flats(self) -> generic_array:
+        flats = self._aux_data.get_flats(self.is_gpu)
+        if flats is None:
+            flats = self._empty_aux_array()
+        return flats
+
+    @flats.setter
+    def flats(self, flats: generic_array):
+        self._aux_data.set_flats(flats)
+
+    # alias
+    @property
+    def flat(self) -> generic_array:
+        return self.flats
+
+    @flat.setter
+    def flat(self, flats: generic_array):
+        self.flats = flats
+
+    @property
+    def shape(self) -> Tuple[int, int, int]:
+        """Shape of the data in this block"""
+        return make_3d_shape_from_array(self._data)
+
+    def to_gpu(self):
+        if not gpu_enabled:
+            raise ValueError("no GPU available")
+        self._data = xp.asarray(self.data, order="C")
+
+    def to_cpu(self):
+        if not gpu_enabled:
+            return
+        self._data = xp.asnumpy(self.data, order="C")
+
+    @property
+    def is_gpu(self) -> bool:
+        return not self.is_cpu
+
+    @property
+    def is_cpu(self) -> bool:
+        return getattr(self._data, "device", None) is None