Ultra l1b extended events

[laspsandoval]

Change Summary

Overview

All of l1b de except any SPICE utilization. Note that I still don't have recent lookup tables so I will update this code once I have them.

New Files

lookup_utils.py

• Contains tools for lookup tables for l1b.

ultra_l1b_extended.py

• Calculates extended raw events for Ultra l1b.

/lookup_tables (5 files)

imap_processing/tests/ultra/test_data/l0/FM45_40P_Phi28p5_BeamCal_LinearScan_phi28.50_theta-0.00_Ultra_Image_Raw_Event_20240207T102746_withFSWcalcs.csv

• test data

Updated Files

imap_ultra_l1b_variable_attrs.yaml

• Added and improved variable descriptions.

conftest.py

• Added some test data.

de.py

• Added direct event data.

Testing

test_lookup_utils.py

• Tests lookup_utils.py

test_ultra_l1b_extended.py

• Tests ultra_l1b_extended.py

test_ultra_l1b.py

• Tests ultra_l1b.py

Environment

poetry.lock
pyproject.toml

[subagonsouth]

Great to see some algorithm code being implemented.

This is a very large PR. I was unable to to a thorough review but provided what feedback I could.

[greglucas]

This is too much for me to review in one sitting, but I left some general comments now.

I'd really like to see this broken up into roughly each individual function as a PR, which means this one PR turns into ~15-20. I think this would go much faster and easier for getting feedback on as well.

[greglucas]

Here and elsewhere in the tests it would be helpful to use the numpy testing utilities for array comparisons.
https://numpy.org/doc/stable/reference/generated/numpy.testing.assert_allclose.html

np.testing.assert_allclose(xc, test_xc.values, rtol=...)

[greglucas]

This seems like it should have an xfail/skip associated with it. A TODO in the tests doesn't seem ideal as I'm not sure we'll get back to it, so it could get lost in time.

Suggested change

	# TODO: the first value doesn't match. Look into this.
	assert np.array_equal(test_energy[1::], energy[1::].astype(float))
	# TODO: the first value doesn't match. Look into this.
	np.testing.assert_array_equal(test_energy[1::], energy[1::].astype(float))

[greglucas]

Should STOP_TYPE be an Enum? What do 1 and 2 mean here?

[greglucas]

You only want the first occurrence, or do you want all sets of indices matching this condition?
Numpy also suggests using nonzero() instead to get the indices. https://numpy.org/doc/stable/reference/generated/numpy.where.html

ph_indices = np.nonzero(de_dataset["STOP_TYPE"] == 1) | (de_dataset["STOP_TYPE"] == 2)

[greglucas]

Suggest changing all "where" to nonzero and making sure you only want the first occurrence or if you want all of the indices. (indices to me means there could be multiples whereas you're only grabbing the first ssd_index currently)

Suggested change

	ssd_indices = np.where(de_dataset["STOP_TYPE"] >= 8)[0]
	ssd_indices = np.nonzero(de_dataset["STOP_TYPE"] >= 8)

[greglucas]

This should be left as "0.0.0" and not updated, see the comment above. There is something going wrong with your install if this is getting committed I think.

[greglucas]

Would suggest an Enum for this. STOP_TYPE.Top, STOP_TYPE.Bottom or similar

[greglucas]

I think there is a lot of duplicated code for the two cases here, so would it make sense to try and factor some of this out? I think you could also subset your dataset instead of keeping track of the indices and using those all the itme.
https://docs.xarray.dev/en/v2024.06.0/generated/xarray.Dataset.where.html

top_ds = de_dataset.where(de_dataset["COIN_TYPE"] == 1)

[laspsandoval]

This is too much for me to review in one sitting, but I left some general comments now.

I'd really like to see this broken up into roughly each individual function as a PR, which means this one PR turns into ~15-20. I think this would go much faster and easier for getting feedback on as well.

Ok. I'll create separate branches to break it up.