v4.2

Savu Version 4.2

Core

• Statistics class:
  • Stats relating to the data calculated alongside each plugin.
  • Can be accessed in Savu for use in plugins.
  • Collated and saved as a table, graphs and a .h5 file for inspection after a run (found in the 'stats' folder).
  • Use option --stats and parse arguments on or off to set stats on or off for a run (on by defualt).
  • MinAndMax plugin is now deprecated (use statistics class instead).
• Iterative plugins (A capability to enable some plugins to be iterative):
  • iterate command to enable control over iterative plugins (see iterate -h for help)

Existing Plugins

Alignment

• For the case of projection_2d_alignment creating an output dataset that
  contains the shifted projections, preview information about this dataset
  is provided to the experimental metadata.

Centering

• Modify vo_centering to enable its use in the iterative alignment
  implementation which shifts the projections.

Simulation

• A sub-pixel misalignment simulation for projections from TomoPhantom
• Various Tomoloader fixes. Closing of the linked nxs file with MPI fixed.

Reconstruction

• AstraReconGPU, 3D GPU methods are added (BP3D_CUDA, CGLS3D_CUDA, SIRT3D_CUDA)
• FBP3D_CUDA method added (filtering before backprojection with BP3D_CUDA)
• ForwardProjector works with the 3D geometry
• 3D geometries can accept metadata for x-y shifts and correct the misalignment
• ToMoBAR (3d version) has got different methods (FBP3D, CGLS3D, FISTA3D) working well with iterative alignment
• GPU memory usage check for tomobar_recon_3D plugin to avoid CUDA error
• tomobar_recon_3D access to regularisation using Wavelets, try set regularisation method e.g. to 'PD_TV_WAVELETS'
• SWLS, PWLS methods for data fidelities are enabled in tomobar_recon_3D
• GPU device indices are controlled through ToMoBAR iterative methods and regularisation

Filters

• GPU memory usage check for ccpi_denoising_gpu_3D plugin to avoid CUDA error
• DezingerSinogram changed to utilise new statistics class.
• DownsampleFiler changed to utilise new statistics class.

Savers

• ImageSaver changed to utilise new statistics class.

New plugins

Alignment

• projection_2d_alignment - works with 2 sets of 3D projection data by comparing projection images and estimating vertical-horizontal shifts in data. The vector shifts then stored in experimental data to be used later in 3D vector geometry.

Filters

• wavelet_denoising_gpu - a GPU plugin for denoising using Wavelets. Highly optimised for GPU performance.

Corrections

• phase_unwrapping - a plugin for unwrapping phase-retrieved images

Centering

• 360_centering - a plugin to calculate centre of rotation.

Updated and new packages as dependencies

• A new pypwt GPU wavelet package added through Jenkins build and savu-dep channel
• ToMoBAR and TomoPhantom packages have been updated

Configurator

• Allow a list as a single dimension input to the preview parameter
• Allow a start keyword inside the preview parameter
• Allow a parameter to be a directory path within the Savu folder
• Asterix line added to indicate the start and end of the process list
• Line separator added to indicate an iterative plugin loop
• Include a link to relevant online plugin guides
• Warning added when loading plugins from a user directory
• First dataset passed by default to the next plugin

New Commands

• savu_mod - a way to modify one parameter present in a plugin list.
• mod -g parameter - a way to modify a parameter in the process list globally (for all plugins)

Documentation

• Plugin API moved to dropdown boxes on every plugin documentation page
• Plugin template links updated
• Video demos added

BUGS

• res_norm bug when using AstaReconGPU with CGLS_CUDA
• Fix indentation for the plugin_generator command
• Allow saving to inner plugin directories when using the plugin_generator command
• Plugin number and order error fixed

Other

• The test dataset 24737.nxs has been changed to tomo-standard.nxs
• The synthetic test data has been added
• Environment variable type is replaced with GPUarch_nodes
• savu_mod - Modify one parameter present in a plugin list.
• Save the job command to a log file
• Save the directory the command was run to a log file

Version 4.0

Core Updates:

• Adds the PluginTools base class from which all *_tools.py files inherit
  • Contains 3 classes
    • PluginParameters
      • Separates all parameter methods from plugin.py
      • Performs checks on parameter definitions
      • Organises parameters with dependencies
    • PluginCitations
      • Performs citation information handling
    • PluginDocumentation
• Automatically outputs all citations to citations.txt in output folder
• Outputs Savu exceptions to the user log
• Fixes bug in data padding when step > 1
• Updates inheritance of slice list classes

Plugin Updates:

• Splits plugins into processing and "tools" (parameter and citation) parts
• Adds a *_tools.py file for every plugin
  • Updates parameter definitions for all plugins
• Adds citations for regularisation methods
• Updates LFOVLoader and ImageStitching plugins
• Fixes convert360180sinogram
• Adds extra options to downsample_filter plugin
• Cleans out unused plugins into unregistered folder
• Fixes a bug in MinAndMax plugin
• Adds pattern checking to ImageSaver

Configurator Updates:

• Adds new parameter_utils.py class
  • Performs type checks on all parameters set in the configurator
  • Checks for all basic types, e.g, int, float, bool, filepath
  • Checks for complex types, e.g. list combinations, dictionaries, preview
• Hides obsolete parameters for a fixed dependency
• Fixes the order of plugin parameters
• Adds visibility levels of basic, intermediate, advanced
• Adds option to view only dataset fields
• Adds cite command
• Adds options list
• Adds expand command for simplified previewing notation
• Saves plugin citations to process list nexus file

Documentation Updates:

• Sphinx and rst updates
• Savu logo and reference fixes
• Ports Savu confluence pages to docs
• Adds example plugin templates
• Fixes plugin API
• Updates installation information
• Adds citation information
• New pages on how to develop a plugin and define parameters

Other Updates:

• Adds savu-lite installation for PC
• A more flexible and accelerated installer for Savu HPC
• Ports continuous integration tests to Github Actions
• Fixes Savu profiler
• Fixes Savu citation extractor
• Adds webhooks/event manager to Github repo for better coding practices
• Expands test coverage of plugins
• Local Jenkins server utilised to speed up the release process:
  • A new fully automated software release system for Savu using Jenkins server
  • Nightly builds for dependencies
  • Nightly builds for Savu-lite using pre-built dependencies
  • Bash scripts to automatically update packages in conda environment
  • Semi-automated process of updating packages in Savu HPC installation

New Plugins Added:

• filters:
  • inpainting:
    • inpainting.py
• reconstructions:
  • projectors:
    • forward_projector_cpu.py
    • forward_projector_gpu.py

Version 3.0

Core updates:

• Ports Savu to Python 3
• Fixes dosna transport
• Restructures experiment collection and plugin runner to reduce the number of plugin load repetitions
• Fixes 4D processing problems, including chunks not set
• Fixes incorrect dimension error when fixing nFrames to be an integer
• Fixes error in choosing first slice dimension
• Fixes incorrect nxentry group name
• Adds VOLUME_3D pattern
• Adds sinomovie pattern
• Fixes volume patterns for data with no y axis (single entry)
• Adds option to "getall" of a pattern of lower dimensions than the one requested

Plugin updates:

• First data simulation plugin via TomoPhantom integration
• Faster Astra toolbox reconstruction using vector geometry
• GPU implementations of median_filter and dezinger plugins
• Large field of view experiment integration
• Improves MinAndMax plugin used for ImageSaver
• Improves ImageSaver: save 8-bit, 16-bit tiff; manually select global min-max
• Fixes DarkFlatFieldCorrection SINOGRAM processing
• Updates the Fresnel filter for projection or sinogram processing
• Adds novel modifications to tomobar plugins fully based on vector Astra geometry
• Fixes error in base_recon when previewing reconstruction methods
• Mrc loader update
• Fixes nxtomo_loader not using image_key_path parameter
• Allows user to set a value of the outer circle mask in both Tomopy_recon and Astra_recon
• Fixes error in base_recon when previewing reconstruction methods
• Fixes incorrect slicing in mipmap plugin
• Correction to faulty flats averaging
• Adds getall flag option to plugin_data

Configurator updates:

• Adds error message when loading a module that exists but failed on import
• Replaces Gnureadline with readline
• Adds examples flag option to include plugin templates
• Amends mutations to check process list version number
• Adds option to output error messages when loading modules via -e flag

Other updates:

• Updates profiler
• Adds iterative plugin templates
• Adds savu-lite conda installation for local PC
• Adds savu tools to easily extract parameter information from process lists and loader yaml templates
• Adds zocalo qsub and new JSV to fix variable nodes error in launcher script
• Adds keep flag to launcher script for option to not delete data when passing an intermediate folder
• Improves tests
• Refines savu installer

New Plugins added:

• basic_operations:
  • data_threshold.py
  • elementwise_arrays_arithmetics.py
  • value_mask_replacement.py
• corrections:
  • mtf_deconvolution.py
• filters:
  • denoising:
    • ccpi_denoising_cpu.py
    • ccpi_denoising_cpu_3D.py
    • ccpi_denoising_gpu.py
    • ccpi_denoising_gpu_3D.py
• loaders:
  • full_field_loaders:
    • lfov_loader.py
• reshape:
  • image_stitching.py
• ring_removal:
  • ring_removal_interpolation.py
• segmentation:
  • evolving_contours:
    • region_grow.py
    • region_grow3D.py
  • geo_distance:
    • geo_distance3D.py
  • morphological_operations:
    • morph_proc_line3D.py
    • morph_remove_objects.py
• simulation:
  • tomo_phantom.py
  • tomo_phantom_quantification.py

Plugin Replacements:

• dezinger => dezinger, dezinger_gpu
• median_filter => median_filter, median_filter_gp
• AstraReconGpu (new vectorised geometry)

Deprecated plugins:

• dezinger
• dezinger_simple
• dezinger_sinogram
• median_filter

Removed plugins:

• base_fluo_fitter.py
• fastxrf_fitting.py
• simple_fit_xrf.py
• base_i18_multi_modal_loader.py
• i18_fluo_loader.py
• i18_mm_loader.py
• i18_monitor_loader.py
• i18_stxm_loader.py
• i18_xrd_loader.py

Version 2.4

Core updates:

• Adds singularity recipe and conda recipes for hdf5, h5py, openmpi and fftw
• Reverts axis labels to correct length after temporary reduction of frames (plugin previewing)
• Updates astra install to dev version
• Fixes incorrect numbering in process lists when nplugins > 9
• Stops GPU check from failing when no GPU processes are requested
• Converts data transfer bounds from frames to bytes
• Speed improvements:
  • Allows a wider variety of mft for data with dim > 3
  • Improves mft calculations for data with multiple slice dimensions
  • Improves chunking for > 3D datasets
• Allows input OR output datasets with different slice lengths
• Fixes inconsistent mft for in/out datasets with different dtypes
• Resets process frames counter for compatibility with parameter tuning
• dls cluster launcher script updates:
  • Adds auto-deletion of intermediate files when tmp directory is passed to dls launcher script
  • Rewrites savu_launcher.sh for compatibility with Science cluster and Hamilton
  • Integrates developer launcher script with standard launcher script

Plugin updates:

• Simplifies accessing current frame index from plugins
• Fixes initial volume in reconstruction
• Fixes initial volume in reconstruction
• Fixes problem in yaml converter when "savu" is in the file path
• Fixes but when outputting and reloading dictionary entries in meta data
• Fixes mrc loader to be compatible with checkpointing
• Fixes dark and flat previewing for tomo data with no image key
• Reconstruction COR updates:
  • Allows manual input as COR (as dict)
  • Allows COR as dataset
• Updates citation information for DistortionCorrection plugin
• Replaces C-wrapped DistortionCorrection plugin with python implemention
• Corrects the use of angles parameters in CameraRotCorrection to allow for parameter tuning
• Allows different angles per scan in NxtomoLoader
• Allows ImageLoader without dark field data
• Further improvements to auto-centering method

New plugins added:

• basic_operations:
  • arithmetic_operations.py
  • data_rescale.py
  • get_data_statistics.py
• corrections:
  • convert_360_180_sinogram.py
• filters:
  • ccpi_denoising_cpu.py
  • ccpi_denoising_gpu.py
  • fresnel_filter.py
  • hilbert_filter.py
• reconstructions:
  • tomobar_recon.py
  • tomobar_recon_3D.py
  • tomobar_recon_cpu.py
  • visual_hulls_recon.py
• reshape:
  • mipmap.py
• ring_removal:
  • remove_all_rings.py
  • remove_large_rings.py
  • remove_unresponsive_and_fluctuating_rings.py
  • ring_removal_filtering.py
  • ring_removal_fitting.py
  • ring_removal_regularization.py
  • ring_removal_sorting.py
• savers:
  • image_saver.py
• stats:
  • min_and_max.py
• visualisation:
  • ortho_slice.py
• segmentation:
  • geo_distance.py
  • gmm_segment3D.py
  • mask_evolve.py
  • mask_initialiser.py
  • morph_snakes.py
  • morph_snakes3D.py
  • morphological_operations/morph_proc.py
  • thresh_segm.py
  • i23segmentation:
    • final_segment_i23.py
    • i23_segment.py
    • i23_segment3D.py

v2.3.2

This release tests a new version of astra (1.9dev) with Savu

Version 2.3.1

Patch for version 2.3 to update installation scripts and software versions. The installation process in version 2.3 causes multi-node MPI jobs to hang, due to processes failing and the remainder waiting at a barrier.

Version 2.3

Core updates:

• Adds checkpointing
  • With option to change number of processes at a plugin checkpoint level
• Moves all system parameters to a separate system_parameters yaml file
• Removes the hdf5 chunk cache (by setting to zero in system parameters file)
  • Significantly improves speed
• Refines non-hdf5 datatypes to allow re-creation from output file and better copying
• Adds "all_versions" file for mapping version numbers to installer zip files
• Refines installer script
• Restructures mpi folder and renames to system_files
• Configurator:
  • Adds option of dictionary as plugin parameter
  • Removes unnecessary quotes around strings when re-loading and displaying the plugin list
  • Improves error output messages

Plugin updates:

• Replaces SavuLoader with SavuNexusLoader
• Adds SavuNexusLoader
  • Allows re-loading of multiple Savu output datasets from a Savu output Nexus file
• Adds DezingerSinogram plugin
• Adds SubPixelShift plugin
• Adds RingRemovalNormalization plugin
• Fixes MultiNxtomoLoader
• Fixes plugin previewing for dimensions greater than 3
• Allows image key and dezinger with no darks and flats
• Enhances Hdf5LoaderTemplate
  • Adds multiple dataset stack and concatenate options to
  • Adds wildcard keyword to access the wildcard values when data stitching in Hdf5TemplateLoader
  • Adds option to remove dimensions in StitchData type
• Fixes data with no angles in NxtomoLoader
• Refines Tomopy
  • Improves outer_pad and removes sino centering
• Amends log function in base recon and exposes the function as a parameter
• Corrects incorrect citation information in CcpiRingArtefactFilter
• Adds plugin example templates

Version 2.2.1

Core updates:

• Improves dockerfile for external use
• Improves HDF5 data chunking, with significant speed increases

Plugin updates:

• Amends DistortionCorrection parameters
  • as agreed and tested with DLS beamline staff
• Adds 'multiple' frames request with a maximum limit
• DezingFilter renamed as DezingerSimple
• Adds Dezinger plugin
  • Adds updated version of Robert Atwood's C-implemented algorithm back in to Savu
  • This is a faster more accurate implementation
• Updates to DezingerSimple
  • Maximum 8 frames now requested due to memory issues
  • Darks and flats now only processed 8 at a time

Version 2.2

Core Updates:

• Adds Savu runtime template option:
  • Adds syntax in configurator to choose subset of parameters
  • Adds option to create a template (yaml file) when saving process list
  • Adds option to pass template at runtime
  • Adds updates to process list in savu from template file
• Adds Dosna (Distributed Object Store Numpy Array) transport
  • Option to replace hdf5 backend
• Adds dockerfile
  • Updates savu_installer with --no_prompts and --PREFIX options
• Fixes bug of multiple processing accessing the same chunk
  • wrong value updating max_frames_transfer dict entry for use in chunking calculations
  • should improve performance
• Separates transfer of dark and flat frames that are far apart in data with image key
  • Significantly improves performance as hdf5 doesn't handle this well

Plugin Updates:

• Adds template loaders that parse a data description from a yaml file
  • Adds hdf5_template_loader
  • Adds image_template_loader
  • Adds folder of template examples
  • Allows greater flexibility in data loading and clearer descriptions
• Adds camera_rot_correction plugin
  • Align data with tilted cor
• Refines nexus multi-modal loaders
• Updates tiff saver
  • Change to output folder name
• Updates Fabio data type (used in image loader):
  • Allows stepping over data
  • Allows stitching in any dimensions
  • Sets datatype to 32 bit
• Refines image_loader
• Fixes vo_centering for start_pixel values when downsampling is applied to the data
• Updates distortion correction parameters

Version 2.1.1

Updates installation scripts