Reduction process: effective-instrument geometry.

[ekapadi]

Description of work

At the end of the reduction process, the instrument associated with each output workspace is modified. A new effective instrument is substituted for the instrument attached to each workspace. This instrument has the same number of pixels as there are group-ids, and the location of each pixel is set to the mean location of the unmasked original pixels participating in that pixel group. By implication, this substitution results in there being one pixel per spectrum in each output workspace. (An exception to this substitution is the combined pixel-mask workspace, which is not modified.)

Explanation of work

This commit includes the following changes:

• A new EffectiveInstrumentRecipe implemented as a subrecipe called by ReductionRecipe for each grouping;

• Modification of ReductionIngredients to include the unmasked PixelGroups;

• Modification of SousChef.prepReductionIngredients to include the unmasked PixelGroups;

• Modification of existing unit tests, and implementation of new unit tests to verify the new subrecipe's execution.

A known issue: EditInstrumentGeometry deletes, but does not regenerate the XML associated with the newly-edited instrument. Unfortunately, this tends to generate a lot of misleading log messages when the associated workspaces are saved, and then reloaded. This isn't so much an issue with EditInstrumentGeometry specifically, but rather with the Mantid instrument framework.

To test

Existing unit tests have been modified, and new unit tests have been added to cover the new subrecipe. A new CIS-test script has been added at tests/cis_tests/effective_instrument_script.py.

Dev testing

In addition to running the unit tests, and the CIS-test script, the GUI-panels should be run for the full set of workflows. There should be no observable behavior changes. (With the exception of the log messages associated with the new output changes, and a warning message associated with the "known issue" indicated above.) Note that you'll also need the newly-saved output from the reduction workflow to test this PR.

The instrument associated with any output workspace can be examined directly. It should now have the same number of pixels as there are spectra in the workspace. Please see (and run through) the new CIS-test script at: tests/cis_tests/effective_instrument_script.py.

CIS testing

The new changes function automatically as part of the reduction process.
The instruments attached to the reduction-output workspaces should be examined, and verified to be as expected.
For comparison purposes, or in the case a work-around is required, the "application.yml" flag reduction.output.useEffectiveInstrument can be used to turn off the effective-instrument substitution.

Link to EWM item

EWM#6549

Verification

• [ x] the author has read the EWM story and acceptance critera
• the reviewer has read the EWM story and acceptance criteria
• the reviewer certifies the acceptance criteria below reflect the criteria in EWM

Acceptance Criteria

1. Where possible, all new changes are covered by unit tests.

2. Reduction output workspaces will have a new effective instrument that has the same number of pixels as there are spectra in the workspace.  (And by assumption, these workspaces will have a number of spectra that correspond to the number of group-ids in the associated grouping.)

3. The name of the effective instrument will be "SNAP_<grouping name>".

4. Locations of each effective pixel will correspond to those of the unmasked PGP values for that pixel's group-id.  Note that tests/cis_tests/effective_instrument_script.py can be used to obtain and verify these values.  Mantid-workbench instrument view might also be a useful way to verify this quickly.

This list is for ease of reference, and does not replace reading the EWM story as part of the review. Verify this list matches the EWM story before reviewing.

• acceptance criterion 1
• acceptance criterion 2
• acceptance criterion 3
• acceptance criterion 4

[ekapadi]

I'm adding a cis-test script to help with verifying acceptance criterion #4.

[ekapadi]

I ran into a snag with the acceptance criteria, so please wait to review this PR. There are some required Mantid changes in order to get the CIS-test to work. :(

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.58%. Comparing base (b628382) to head (52b7143).
Report is 3 commits behind head on next.

Additional details and impacted files

@@            Coverage Diff             @@
##             next     #461      +/-   ##
==========================================
+ Coverage   96.55%   96.58%   +0.03%     
==========================================
  Files          66       67       +1     
  Lines        4960     5010      +50     
==========================================
+ Hits         4789     4839      +50     
  Misses        171      171              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ekapadi]

Everything is now ready for review! :)

[walshmm]

Calibration and Normalization run without issue.
Reduction produces workspaces with the expected instrument name
but I'm not seeing any instrument in the instrument view on completion of reduction

[walshmm]

they reload with the new instrument but still none in the instrument viewer

[walshmm]

(the cis test script did pass with my data subbed in)

[walshmm]

now with the flag off the assert correctly fails when checking to see if a new instrument geometry was applied to the output

However after activating the flag and rerunning the cis test it seems to fail, the instrument loaded on the workspace is None

[walshmm]

after rebuilding my conda env and reruning reduction it seemed to pass

# packages in environment at /SNS/users/wqp/.conda/envs/mamba/envs/SNAPRed:
#
# Name                    Version                   Build  Channel
mantid                    6.11.20241204.1559 py310h7c5db07_0    mantid/label/nightly
mantiddocs                6.11.20241204.1559      h2c57996_0    mantid/label/nightly
mantidqt                  6.11.20241204.1559 py310hb8bf3eb_0    mantid/label/nightly
mantidqtinterfaces        6.11.20241204.1559          pypi_0    pypi
mantidworkbench           6.11.20241204.1559 py310h38c35bf_0    mantid/label/nightly