Various CoDICE L1 Updates #711

bourque · 2024-07-24T17:35:29Z

Change Summary

Overview

This PR contains various updates to the CoDICE L1 algorithm, mainly to clean some things up, add support for lo-pha and hi-pha data products (event data), and changes required to get recently-acquired simulated data working in the pipeline.

Updated Files

imap_processing/cdf/config/imap_codice_global_cdf_attrs.yaml
- Added CDF attrs for event data products
imap_processing/cdf/config/imap_codice_l1a_variable_attrs.yaml
- Formatting improvements, added missing attributes that are now required
imap_processing/cdf/config/imap_codice_l1b_variable_attrs.yaml
- Formatting improvements, added missing attributes that are now required
imap_processing/codice/codice_l0.py
- Added XTCE mapping for event data
imap_processing/codice/codice_l1a.py
- Added support for event data; moved housekeeping dataset creation into module to be consistent with where similar code is being defined; other various cleanup/organizing
imap_processing/codice/constants.py
- Moved hard-coded APID list from codice_l1a to constants file
imap_processing/codice/packet_definitions/*.xml
- Updates from newer version of space_packet_parser and updates packet definition spreadsheet
imap_processing/codice/utils.py
- Moved create_hskp_dataset to codice_l1a.py to be consistent with where the code to create other datasets resides
imap_processing/tests/codice/data/imap_codice_l0_*.pkts
- Updated/new simulated data
imap_processing/tests/codice/test_codice_l0.py
- Renamed functions with hskp to be consistent with naming convention
imap_processing/tests/codice/test_codice_*.py
- Improved formatting and updated tests

…how science datasets are created

bourque · 2024-07-24T17:49:15Z

imap_processing/cdf/config/imap_codice_l1a_variable_attrs.yaml

+    CATDESC: ""
+    DISPLAY_TYPE: no_plot
+    FIELDNAM: ""
+    FILLVAL: -9223372036854775808
+    FORMAT: I12
+    LABLAXIS: ""
+    REFERENCE_POSITION: ""
+    RESOLUTION: ""
+    SCALETYP: linear
+    TIME_BASE: ""
+    TIME_SCALE: ""
+    UNITS: dN
+    VALIDMIN: -9223372036854775808
+    VALIDMAX: 9223372036854775807
+    VAR_TYPE: data


This was just to have consistent indenting across the file.

bourque · 2024-07-24T17:51:18Z

imap_processing/cdf/config/imap_codice_l1a_variable_attrs.yaml

+    DISPLAY_TYPE: ""
    FIELDNAM: Energy step
+    FILLVAL: ""
    FORMAT: A3
+    LABLAXIS: ""
+    REFERENCE_POSITION: ""
+    RESOLUTION: ""
+    TIME_BASE: ""
+    TIME_SCALE: ""
+    UNITS: ""
+    VALIDMAX: ""
+    VALIDMIN: ""


I found that I needed to add these in order to pass the new check here:

imap_processing/imap_processing/cdf/cdf_attribute_manager.py

Line 309 in a257b7e

# Validating required schema

Hmm, is that a bug in the validation code? Based on the schema YAML here, I wouldn't expect all of these additions to be necessary.

imap_processing/imap_processing/cdf/config/shared/default_variable_cdf_attrs_schema.yaml

Line 461 in a257b7e

metadata:

If I am understanding correctly, this is checking against whatever is defined as "required" in the default_variable_cdf_attrs_schema.yaml file. It loos like all of these are indeed required according to this file, for example TIME_BASE:

attribute_key: # ===== EPOCH-ONLY VARIABLE ATTRIBUTES ===== TIME_BASE: description: > fixed (0AD, 1900, 1970 (POSIX), J2000 (used by CDF_TIME_TT2000), 4714 BC (Julian)) or flexible (provider-defined) required: true # NOT Required in ISTP Guide

Right. I am wondering if it should only be checking what is required and is present in the list of attributes defined for the datatype. Not important to this PR though.

Based on slack convos, sounds like Maxine will fix this eventually. I will hang tight on merging this PR and update accordingly when the fix is in.

imap_processing/codice/codice_l1a.py

imap_processing/codice/packet_definitions/P_COD_HI_INST_COUNTS_AGGREGATED.xml

subagonsouth

This mostly looks good to me. I didn't review the XTCE changes.

subagonsouth · 2024-07-24T20:30:57Z

imap_processing/cdf/config/imap_codice_l1a_variable_attrs.yaml

+    DISPLAY_TYPE: ""
    FIELDNAM: Energy step
+    FILLVAL: ""
    FORMAT: A3
+    LABLAXIS: ""
+    REFERENCE_POSITION: ""
+    RESOLUTION: ""
+    TIME_BASE: ""
+    TIME_SCALE: ""
+    UNITS: ""
+    VALIDMAX: ""
+    VALIDMIN: ""


Hmm, is that a bug in the validation code? Based on the schema YAML here, I wouldn't expect all of these additions to be necessary.

imap_processing/imap_processing/cdf/config/shared/default_variable_cdf_attrs_schema.yaml

Line 461 in a257b7e

metadata:

imap_processing/codice/codice_l1a.py

imap_processing/tests/codice/test_codice_l1a.py

imap_processing/codice/codice_l1a.py

greglucas

Minor comments is all. There are a lot of refactor things going on, but no new processing from what I could tell. If you could break up the yaml/metadata additions and code refactor in future PRs I think that would be helpful too.

I am hopeful that going to a combined XTCE with dynamic variable length and starting from the xarray datasets will be helpful for you here.

Note that we aren't including the short/long description in the housekeeping datasets automatically and planning on using yaml configuration files for all metadata.
See this thread for more discussion: #687 (comment)

greglucas · 2024-07-26T14:26:02Z

imap_processing/tests/codice/test_codice_l1a.py

+
+    # Write the dataset to a CDF so it can be manually inspected as well
+    file_path = write_cdf(dataset)
+    print(f"CDF file written to {file_path}")


Suggested change

print(f"CDF file written to {file_path}")

I suppose I should change this to a logging statement. I like to have this path printed so that I can copy over the resulting CDF in case I want to manually inspect it.

imap_processing/tests/codice/test_codice_l1a.py

bourque · 2024-07-26T20:18:04Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

…g into codice-l1-updates

bourque · 2024-08-14T21:49:42Z

I added a TODO in the code to remind myself that the CDF variable attributes config needs to be updated once the problem with the cdf_attribute_manager checks is resolved. I am going to merge this now so that I can start working off of this code to make more updates.

bourque · 2024-08-14T22:23:06Z

pre-commit.ci autofix

bourque added 14 commits July 17, 2024 14:32

Moved create_hskp_dataset to within l1a module to be consistent with …

c7ecb63

…how science datasets are created

Now using epoch attrs defined in yaml files instead of dataclasses

0155067

Fixed warnings given off from CdfAttributeManager

f5ca632

Improvements to some CDF attribute definitions

a78ddb6

Renamed method for condifuring data products

adffee2

Added initial support for lo and hi event data processing

391fcc6

Improved formatting for readability

3fe12ed

Changes to support actual hi-sectored simulated data

42e347b

Changes to support actual hi-omni simulated data

ff00df8

Changes to support actual hi-counters simulated data

91da446

Simplified the code for event/hskp/science processing

37f7ab9

Improved readability

ec810a0

Added placeholder for HI_PHA_NAMES

cbe0b0a

Removed PHA variable name placeholders

3811061

bourque added Ins: CoDICE Related to the CoDICE instrument Level: L1 Level 1 processing labels Jul 24, 2024

bourque self-assigned this Jul 24, 2024

bourque commented Jul 24, 2024

View reviewed changes

imap_processing/codice/codice_l1a.py Show resolved Hide resolved

bourque commented Jul 24, 2024

View reviewed changes

imap_processing/codice/packet_definitions/P_COD_HI_INST_COUNTS_AGGREGATED.xml Show resolved Hide resolved

bourque requested review from a team, greglucas, subagonsouth and tech3371 and removed request for a team July 24, 2024 17:55

subagonsouth reviewed Jul 24, 2024

View reviewed changes

greglucas approved these changes Jul 26, 2024

View reviewed changes

Merge branch 'dev' into codice-l1-updates

b799411

pre-commit-ci bot and others added 6 commits July 26, 2024 20:18

[pre-commit.ci] auto fixes from pre-commit.com hooks

89d3bbd

for more information, see https://pre-commit.ci

Moved hard-coded test file paths to conftest

18962a2

Merge branch 'codice-l1-updates' of github.com:bourque/imap_processin…

89574db

…g into codice-l1-updates

Removed print statement in favor of logging

33d9f3a

Added TODO to fix atttributes in the future

1788bef

Merge branch 'dev' into codice-l1-updates

72b28dc

Fixed bug in epoch attrs name

553c220

bourque added 3 commits August 15, 2024 11:33

Fixed mypy error

3cb7736

Fixed mypy error

e7f44c9

Fixed mypy error

d8e057b

bourque merged commit b3d0377 into IMAP-Science-Operations-Center:dev Aug 15, 2024
17 checks passed

bourque deleted the codice-l1-updates branch August 15, 2024 18:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various CoDICE L1 Updates #711

Various CoDICE L1 Updates #711

bourque commented Jul 24, 2024 •

edited

Loading

bourque Jul 24, 2024

bourque Jul 24, 2024

subagonsouth Jul 24, 2024

bourque Jul 25, 2024

subagonsouth Jul 25, 2024

bourque Jul 26, 2024

subagonsouth left a comment

subagonsouth Jul 24, 2024

greglucas left a comment

greglucas Jul 26, 2024

bourque Jul 26, 2024

bourque commented Jul 26, 2024

bourque commented Aug 14, 2024

bourque commented Aug 14, 2024

Various CoDICE L1 Updates #711

Various CoDICE L1 Updates #711

Conversation

bourque commented Jul 24, 2024 • edited Loading

Change Summary

Overview

Updated Files

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

subagonsouth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greglucas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bourque commented Jul 26, 2024

bourque commented Aug 14, 2024

bourque commented Aug 14, 2024

bourque commented Jul 24, 2024 •

edited

Loading