Ewm8086 remove background by smoothing before cross-correlation

[rboston628]

Description of work

This improves RemoveEventBackground, and renames it to RemoveSmoothedBackground

• it is faster
• it makes use of SmoothDataExcludingPeaks, rather than reimplementing almost the same work
• the data are properly binned without edge effects and without negative values

Explanation of work

It was determined that inside each group, an averaged background can be calculated by averaging all the spectra in that group then using SmoothDataExcludingPeaks to make a group average background.

This background is subtracted from each individual spectra in the group so that only the peaks remain. After the subtraction, all negatives are zeroed out.

To prevent complications from weak pixels, the background is suitably scaled prior to subtraction, with the scale factor determined by the mean values across the averaged and individual spectra. This step is crucial in keeping peaks in pixels with very weak signal.

To test

Dev testing

You will need an otherwise good calibration run with high background. I suggest 58882.

Run without remove background, and at tweak peak take a screen shot of the graphs. Also look at the mask workspace (diffract_consts_mask_58882), and note how many bins are masked.

Run with remove background (note: you may have to clear all the workspaces for this to happen). Compare the new graphs at tweak peak. Notice the lumpy background is gone, with much sharper peaks.

Check masks. Fewer pixels have been masked. Complete the workflow and save.

CIS testing

First, run the script check_remove.py. This tests the background removal algorithm only. Inspect the workspaces peaks_extractDSP_before and peaks_extractDSP_after, and their focused counterparts (+_foc). Make sure everything is as expected.

Next, run the script cc-and-smooth.py. This mirrors SNAPRed's cross-correlation process, including both the group-by-group cross-correlation and the convergence by median offset. This tests three different ways of removing the background. The ones of interest will have the prefix "total" and "peaks". Look at the CC result (+_cc), the offsets, the before and after, etc.

Finally, run diffcal_pixel_diffraction_background_subtraction_script.py. This will run SNAPRed's PixelDiffCalRecipe with background removal turned on. Inspect the results.

Link to EWM item

EWM#8086

Verification

• the author has read the EWM story and acceptance critera
• the reviewer has read the EWM story and acceptance criteria
• the reviewer certifies the acceptance criteria below reflect the criteria in EWM

Acceptance Criteria

This list is for ease of reference, and does not replace reading the EWM story as part of the review. Verify this list matches the EWM story before reviewing.

• The algorithm RemoveSmoothedBackground creates and subtracts a smoothed background form each spectrum
• Any negative numbers from the subtraction are set to zero
• The binning used in the Rebin step should be changed at L59 to self.dBin: float = min([abs(d) for d in dBin])

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.76%. Comparing base (c860358) to head (7f4a20c).
Report is 2 commits behind head on next.

Additional details and impacted files

@@            Coverage Diff             @@
##             next     #505      +/-   ##
==========================================
+ Coverage   95.57%   95.76%   +0.19%     
==========================================
  Files          68       68              
  Lines        5086     5081       -5     
==========================================
+ Hits         4861     4866       +5     
+ Misses        225      215      -10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rboston628]

Note: this line satisfies one of the acceptance criteria

[rboston628]

Note: this line satisfies one of the acceptance criteria

[rboston628]

MERGE AFTER PR #523

[dlcaballero16]

Tested and remove background works correctly.