Fix/verify bad frames removal

[mgalloy]

Verify that bad frames in the new format are removed and handled through the pipeline.

For science images:

• set bad frame in both cameras to NaN

For darks, flats, and calibration images:

• set only what is specified to NaN

Allow entire level 1 extensions to be NaN, but check in level 2 processing to not continue to level 2 for that file.

Waves files could potentially be repaired when we do the waves analysis.

Tasks

• verify that bad frames are removed in the UCOMP_READ_RAW_DATA routine when specified in the new format
• verify that extensions with NaNs are averaged correctly
• verify that demodulation propagates NaNs correctly
• GBU test for missing the wrong wavelengths

[mgalloy]

Confirmed that UCOMP_READ_RAW_DATA is reading and marking out the bad frames correctly as described at the top of the issue, i.e., both cameras for science extensions and just the listed camera for cal, dark, and flat extensions.

[mgalloy]

I ran 20240409 (just 1079 files) with intermediate products so I can track the NaNs through the pipeline steps. On this day, 1079 has the following bad frames:

20240409.180009.92.ucomp.1079.l0.fts,       0,      31,       1
20240409.183204.79.ucomp.1079.l0.fts,       0,      28,       1
20240409.191146.59.ucomp.1079.l0.fts,       0,      62,       1

Tracking 20240409.180009.92.ucomp.1079.l0.fts, it had 40 extensions:

  1  sci-1079.640-both-rcam            sci    2024-04-09T18:00:10.95  1079.640  rcam
  2  sci-1079.690-both-rcam            sci    2024-04-09T18:00:16.29  1079.690  rcam
  3  sci-1079.800-both-rcam            sci    2024-04-09T18:00:22.63  1079.800  rcam
  4  sci-1079.910-both-rcam            sci    2024-04-09T18:00:28.98  1079.910  rcam
  5  sci-1079.960-both-rcam            sci    2024-04-09T18:00:35.33  1079.960  rcam
  6  sci-1079.640-both-tcam            sci    2024-04-09T18:00:41.68  1079.640  tcam
  7  sci-1079.690-both-tcam            sci    2024-04-09T18:00:48.03  1079.690  tcam
  8  sci-1079.800-both-tcam            sci    2024-04-09T18:00:54.38  1079.800  tcam
  9  sci-1079.910-both-tcam            sci    2024-04-09T18:01:00.73  1079.910  tcam
 10  sci-1079.960-both-tcam            sci    2024-04-09T18:01:07.11  1079.960  tcam
 11  sci-1079.640-both-rcam            sci    2024-04-09T18:01:13.48  1079.640  rcam
 12  sci-1079.690-both-rcam            sci    2024-04-09T18:01:19.83  1079.690  rcam
 13  sci-1079.800-both-rcam            sci    2024-04-09T18:01:26.17  1079.800  rcam
 14  sci-1079.910-both-rcam            sci    2024-04-09T18:01:32.52  1079.910  rcam
 15  sci-1079.960-both-rcam            sci    2024-04-09T18:01:38.87  1079.960  rcam
 16  sci-1079.640-both-tcam            sci    2024-04-09T18:01:45.23  1079.640  tcam
 17  sci-1079.690-both-tcam            sci    2024-04-09T18:01:51.57  1079.690  tcam
 18  sci-1079.800-both-tcam            sci    2024-04-09T18:01:57.94  1079.800  tcam
 19  sci-1079.910-both-tcam            sci    2024-04-09T18:02:04.30  1079.910  tcam
 20  sci-1079.960-both-tcam            sci    2024-04-09T18:02:10.65  1079.960  tcam
 21  sci-1079.640-both-rcam            sci    2024-04-09T18:02:17.00  1079.640  rcam
 22  sci-1079.690-both-rcam            sci    2024-04-09T18:02:23.35  1079.690  rcam
 23  sci-1079.800-both-rcam            sci    2024-04-09T18:02:29.69  1079.800  rcam
 24  sci-1079.910-both-rcam            sci    2024-04-09T18:02:36.06  1079.910  rcam
 25  sci-1079.960-both-rcam            sci    2024-04-09T18:02:42.40  1079.960  rcam
 26  sci-1079.640-both-tcam            sci    2024-04-09T18:02:48.75  1079.640  tcam
 27  sci-1079.690-both-tcam            sci    2024-04-09T18:02:55.10  1079.690  tcam
 28  sci-1079.800-both-tcam            sci    2024-04-09T18:03:01.44  1079.800  tcam
 29  sci-1079.910-both-tcam            sci    2024-04-09T18:03:07.79  1079.910  tcam
 30  sci-1079.960-both-tcam            sci    2024-04-09T18:03:14.13  1079.960  tcam
 31  sci-1079.640-both-rcam            sci    2024-04-09T18:03:20.49  1079.640  rcam (bad RCAM pol state 1)
 32  sci-1079.690-both-rcam            sci    2024-04-09T18:03:26.84  1079.690  rcam
 33  sci-1079.800-both-rcam            sci    2024-04-09T18:03:33.18  1079.800  rcam
 34  sci-1079.910-both-rcam            sci    2024-04-09T18:03:39.55  1079.910  rcam
 35  sci-1079.960-both-rcam            sci    2024-04-09T18:03:45.90  1079.960  rcam
 36  sci-1079.640-both-tcam            sci    2024-04-09T18:03:52.24  1079.640  tcam
 37  sci-1079.690-both-tcam            sci    2024-04-09T18:03:58.59  1079.690  tcam
 38  sci-1079.800-both-tcam            sci    2024-04-09T18:04:04.94  1079.800  tcam
 39  sci-1079.910-both-tcam            sci    2024-04-09T18:04:11.28  1079.910  tcam
 40  sci-1079.960-both-tcam            sci    2024-04-09T18:04:17.63  1079.960  tcam

After averaging like extensions it has 10 extensions:

  1  Corona Stokes IQUV [1079.640 nm]  sci    2024-04-09T18:00:10.95  1079.640  rcam (missing from this average)
  2  Corona Stokes IQUV [1079.690 nm]  sci    2024-04-09T18:00:16.29  1079.690  rcam
  3  Corona Stokes IQUV [1079.800 nm]  sci    2024-04-09T18:00:22.63  1079.800  rcam
  4  Corona Stokes IQUV [1079.910 nm]  sci    2024-04-09T18:00:28.98  1079.910  rcam
  5  Corona Stokes IQUV [1079.960 nm]  sci    2024-04-09T18:00:35.33  1079.960  rcam
  6  Corona Stokes IQUV [1079.640 nm]  sci    2024-04-09T18:00:41.68  1079.640  tcam
  7  Corona Stokes IQUV [1079.690 nm]  sci    2024-04-09T18:00:48.03  1079.690  tcam
  8  Corona Stokes IQUV [1079.800 nm]  sci    2024-04-09T18:00:54.38  1079.800  tcam
  9  Corona Stokes IQUV [1079.910 nm]  sci    2024-04-09T18:01:00.73  1079.910  tcam
 10  Corona Stokes IQUV [1079.960 nm]  sci    2024-04-09T18:01:07.11  1079.960  tcam

There is data in pol state 1, cam 0 for ext 1. This is consistent with what should happen. I compared the average of extensions 1, 11, 21, and 31 to extension 1 in the average file and it matches exactly. The processing shouldn't differ from this point on.

I need to check a waves file, or a file with many bad frames, where it should cause the file to not have data in an extension.

[mgalloy]

I can check 20240409.195647.32.ucomp.1074.l0.fts. It has a bad frame:

20240409.195647.32.ucomp.1074.l0.fts,       0,       6,       0

and is a waves file:

20240409.195647.32.ucomp.1074.l0.fts    sci   1074 nm   6 exts   waves

I need to run the 1074 files for the day with intermediate products.

[mgalloy]

I'm going to test on 20220209 — it is definitely the new format. There are definitely waves files with bad frames on this day, e.g., 20220209.195126.67.ucomp.1074.l0.fts.

[mgalloy]

I processed 20220209 as a test. Results are in process.issue-345 on the new development server.