WIP/MNT: Update SWE to use packet_file_to_datasets #802
Conversation
greglucas
added
Ins: SWE
| @@ -202,7 +187,7 @@ def swe_science(decom_data: list, data_version: str) -> xr.Dataset: | |||
| # Add APID to global attrs for following processing steps | |||
| l1a_global_attrs = cdf_attrs.get_global_attributes("imap_swe_l1a_sci") | |||
| # Formatting to string to be complaint with ISTP | |||
| l1a_global_attrs["packet_apid"] = f"{decom_data[0].header['PKT_APID'].raw_value}" | |||
| # l1a_global_attrs["packet_apid"] = SWEAPID.SWE_SCIENCE.value | |||
| ] | ||
| ) | ||
| for var_name, arr in l0_dataset.variables.items(): | ||
| arr.attrs = cdf_attrs.get_variable_attributes(var_name) |
There was a problem hiding this comment.
looks like we got the CDF attrs set properly for remaining variables. Did you want me to look into CDF attrs for another place/case after this PR?
There was a problem hiding this comment.
Thank you for doing this! This was on my todo list.
| shared_keys = set([x.lower() for x in validation_data.keys()]).intersection( | ||
| first_data.keys() | ||
| ) | ||
| # TODO: Why are all the fields not the same between the two |
There was a problem hiding this comment.
I'm curious why all the validation data isn't the same as the dataset we are producing. I would think that our dataset should contain all the values that are in the validation data, but there is a difference between the validation data keys, and the created dataset keys.
These are the variables that are in one or the other, but not both.
{'cem1', 'phverno', 'cem3', 'phapid', 'phgroupf', 'cem2', 'phtype', 'phseqcnt', 'timestamp', 'cem5', 'cem4', 'cem6', 'cem7', 'phdlen', 'phshf'}
There was a problem hiding this comment.
I see. Those are CCSDS header but they called it different than our standard names in XTCE. They called it like this.
PHVERNO, PHTYPE, PHSHF, PHAPID, PHGROUPF, PHSEQCNT, PHDLEN
and remaining are decommed data of SCIENCE_DATA -
CEM1, CEM2, CEM3, CEM4, CEM5, CEM6, CEM7
There was a problem hiding this comment.
OK, would you like me to remove the comment or update it to something else?
There was a problem hiding this comment.
If you can add comment to validate those CEM# data when we get better data, that would be great.
There was a problem hiding this comment.
👍 Sounds good. I just updated the comment.
| assert decom_test_data[2].data["QUARTER_CYCLE"].derived_value == "THIRD" | ||
| assert decom_test_data[3].data["QUARTER_CYCLE"].derived_value == "FORTH" | ||
| np.testing.assert_array_equal( | ||
| decom_test_data.isel(epoch=slice(0, 4))["quarter_cycle"], [0, 1, 2, 3] |
This is switching SWE over to packet_file_to_datasets rather than iterating through all of the packets individually.
greglucas
force-pushed
the
swe-l0-update
branch
from
c70e024
to
26368fa
Compare
greglucas
merged commit 0a9830d
into
IMAP-Science-Operations-Center:dev
Change Summary
Overview
This is switching SWE over to packet_file_to_datasets rather than iterating through all of the packets individually.