Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Various CoDICE L1 Updates #711

Merged
merged 25 commits into from
Aug 15, 2024
Merged

Various CoDICE L1 Updates #711

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
c7ecb63
Moved create_hskp_dataset to within l1a module to be consistent with …
bourque Jul 17, 2024
0155067
Now using epoch attrs defined in yaml files instead of dataclasses
bourque Jul 17, 2024
f5ca632
Fixed warnings given off from CdfAttributeManager
bourque Jul 17, 2024
a78ddb6
Improvements to some CDF attribute definitions
bourque Jul 17, 2024
adffee2
Renamed method for condifuring data products
bourque Jul 18, 2024
391fcc6
Added initial support for lo and hi event data processing
bourque Jul 24, 2024
3fe12ed
Improved formatting for readability
bourque Jul 24, 2024
42e347b
Changes to support actual hi-sectored simulated data
bourque Jul 24, 2024
ff00df8
Changes to support actual hi-omni simulated data
bourque Jul 24, 2024
91da446
Changes to support actual hi-counters simulated data
bourque Jul 24, 2024
37f7ab9
Simplified the code for event/hskp/science processing
bourque Jul 24, 2024
ec810a0
Improved readability
bourque Jul 24, 2024
cbe0b0a
Added placeholder for HI_PHA_NAMES
bourque Jul 24, 2024
3811061
Removed PHA variable name placeholders
bourque Jul 24, 2024
b799411
Merge branch 'dev' into codice-l1-updates
bourque Jul 26, 2024
89d3bbd
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 26, 2024
18962a2
Moved hard-coded test file paths to conftest
bourque Jul 26, 2024
89574db
Merge branch 'codice-l1-updates' of github.com:bourque/imap_processin…
bourque Jul 26, 2024
33d9f3a
Removed print statement in favor of logging
bourque Jul 26, 2024
1788bef
Added TODO to fix atttributes in the future
bourque Aug 14, 2024
72b28dc
Merge branch 'dev' into codice-l1-updates
bourque Aug 14, 2024
553c220
Fixed bug in epoch attrs name
bourque Aug 14, 2024
3cb7736
Fixed mypy error
bourque Aug 15, 2024
e7f44c9
Fixed mypy error
bourque Aug 15, 2024
d8e057b
Fixed mypy error
bourque Aug 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions imap_processing/cdf/config/imap_codice_global_cdf_attrs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,13 @@ imap_codice_l1a_hi_sectored:
Logical_source: imap_codice_l1a_hi-sectored
Logical_source_description: IMAP Mission CoDICE Hi Level-1A Sectored Data.

imap_codice_l1a_hi_pha:
<<: *instrument_base
Data_level: 1A
Data_type: L1A_hi-pha->Level-1A Hi Event Data
Logical_source: imap_codice_l1a_hi-pha
Logical_source_description: IMAP Mission CoDICE Hi Level-1A Event Data.

imap_codice_l1a_lo_counters_aggregated:
<<: *instrument_base
Data_level: 1A
Expand Down Expand Up @@ -105,6 +112,13 @@ imap_codice_l1a_lo_nsw_species:
Logical_source: imap_codice_l1a_lo-nsw-species
Logical_source_description: IMAP Mission CoDICE Lo Level-1A Non-Sunward Species Counts Data.

imap_codice_l1a_lo_pha:
<<: *instrument_base
Data_level: 1A
Data_type: L1A_lo-pha->Level-1A Lo Event Data
Logical_source: imap_codice_l1a_lo-pha
Logical_source_description: IMAP Mission CoDICE Lo Level-1A Event Data.

# L1b
imap_codice_l1b_hskp:
<<: *instrument_base
Expand Down
71 changes: 44 additions & 27 deletions imap_processing/cdf/config/imap_codice_l1a_variable_attrs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,20 @@
# <=== Defaults ===>
default_attrs: &default
# Assumed values for all variable attrs unless overwritten
DISPLAY_TYPE: no_plot
FILLVAL: -9223372036854775808
FORMAT: I12
UNITS: dN
VALIDMIN: -9223372036854775808
VALIDMAX: 9223372036854775807
VAR_TYPE: data
SCALETYP: linear
CATDESC: ""
DISPLAY_TYPE: no_plot
FIELDNAM: ""
FILLVAL: -9223372036854775808
FORMAT: I12
LABLAXIS: ""
REFERENCE_POSITION: ""
RESOLUTION: ""
SCALETYP: linear
TIME_BASE: ""
TIME_SCALE: ""
UNITS: dN
VALIDMIN: -9223372036854775808
VALIDMAX: 9223372036854775807
VAR_TYPE: data
Comment on lines +7 to +21
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was just to have consistent indenting across the file.


codice_support_attrs: &support_default
<<: *default
Expand All @@ -27,41 +33,51 @@ energy_attrs:
VAR_TYPE: support_data

epoch_attrs:
CATDESC: Time, number of nanoseconds since J2000 with leap seconds included
FIELDNAM: epoch
LABLAXIS: epoch
FILLVAL: -9223372036854775808
FORMAT: " " # Supposedly not required, fails in xarray_to_cdf
VALIDMIN: -9223372036854775808
VALIDMAX: 9223372036854775807
UNITS: ns
VAR_TYPE: support_data
SCALETYP: linear
MONOTON: INCREASE
TIME_BASE: J2000
TIME_SCALE: Terrestrial Time
REFERENCE_POSITION: Rotating Earth Geoid
<<: *default
CATDESC: Time, number of nanoseconds since J2000 with leap seconds included
FIELDNAM: epoch
LABLAXIS: epoch
FILLVAL: -9223372036854775808
FORMAT: " " # Supposedly not required, fails in xarray_to_cdf
VALIDMIN: -9223372036854775808
VALIDMAX: 9223372036854775807
UNITS: ns
VAR_TYPE: support_data
SCALETYP: linear
MONOTON: INCREASE
TIME_BASE: J2000
TIME_SCALE: Terrestrial Time
REFERENCE_POSITION: Rotating Earth Geoid

# <=== Labels ===>
energy_label:
CATDESC: Energy per charge (E/q) sweeping step
DISPLAY_TYPE: ""
FIELDNAM: Energy step
FILLVAL: ""
FORMAT: A3
LABLAXIS: ""
REFERENCE_POSITION: ""
RESOLUTION: ""
TIME_BASE: ""
TIME_SCALE: ""
UNITS: ""
VALIDMAX: ""
VALIDMIN: ""
Comment on lines +42 to +53
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found that I needed to add these in order to pass the new check here:

imap_processing/imap_processing/cdf/cdf_attribute_manager.py

Line 309 in a257b7e

# Validating required schema

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, is that a bug in the validation code? Based on the schema YAML here, I wouldn't expect all of these additions to be necessary.

imap_processing/imap_processing/cdf/config/shared/default_variable_cdf_attrs_schema.yaml

Line 461 in a257b7e

metadata:

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I am understanding correctly, this is checking against whatever is defined as "required" in the default_variable_cdf_attrs_schema.yaml file. It loos like all of these are indeed required according to this file, for example TIME_BASE:

attribute_key:
  # ===== EPOCH-ONLY VARIABLE ATTRIBUTES =====
  TIME_BASE:
    description: >
      fixed (0AD, 1900, 1970 (POSIX), J2000 (used by CDF_TIME_TT2000),
      4714 BC (Julian)) or flexible (provider-defined)
    required: true    # NOT Required in ISTP Guide

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. I am wondering if it should only be checking what is required and is present in the list of attributes defined for the datatype. Not important to this PR though.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on slack convos, sounds like Maxine will fix this eventually. I will hang tight on merging this PR and update accordingly when the fix is in.

VAR_TYPE: metadata

# <=== Dataset Attributes ===>
# TODO: resolve ISTP warning: Width of F10 FORMAT may be insufficient for
# values. Adjust FORMAT or VALIDMIN/VALIDMAX value to resolve.
acquisition_times_attrs:
<<: *default
CATDESC: Time of acquisition for the energy step
DEPEND_1: energy
FIELDNAM: Acquisition Time
FILLVAL: 1.7976931348623157e+308
FILLVAL: .NAN
FORMAT: F10.3
LABLAXIS: Acq Time
UNITS: ms
VALIDMIN: 0
VALIDMAX: 1.7976931348623157e+308
VALIDMAX: 1000
VAR_TYPE: support_data

counters_attrs: &counters
Expand All @@ -79,6 +95,7 @@ counters_attrs: &counters
esa_sweep_attrs:
<<: *default
CATDESC: ElectroStatic Analyzer Energy Values
DEPEND_1: energy
FIELDNAM: ESA Voltage
FORMAT: I19
LABLAXIS: ESA V
Expand Down
77 changes: 46 additions & 31 deletions imap_processing/cdf/config/imap_codice_l1b_variable_attrs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,20 @@
# <=== Defaults ===>
default_attrs: &default
# Assumed values for all variable attrs unless overwritten
DISPLAY_TYPE: no_plot
FILLVAL: -9223372036854775808
FORMAT: I12
UNITS: dN
VALIDMIN: -9223372036854775808
VALIDMAX: 9223372036854775807
VAR_TYPE: data
SCALETYP: linear
CATDESC: ""
DISPLAY_TYPE: no_plot
FIELDNAM: ""
FILLVAL: -9223372036854775808
FORMAT: I12
LABLAXIS: ""
REFERENCE_POSITION: ""
RESOLUTION: ""
SCALETYP: linear
TIME_BASE: ""
TIME_SCALE: ""
UNITS: dN
VALIDMIN: -9223372036854775808
VALIDMAX: 9223372036854775807
VAR_TYPE: data

codice_support_attrs: &support_default
<<: *default
Expand All @@ -27,41 +33,51 @@ energy_attrs:
VAR_TYPE: support_data

epoch_attrs:
CATDESC: Time, number of nanoseconds since J2000 with leap seconds included
FIELDNAM: epoch
LABLAXIS: epoch
FILLVAL: -9223372036854775808
FORMAT: " " # Supposedly not required, fails in xarray_to_cdf
VALIDMIN: -9223372036854775808
VALIDMAX: 9223372036854775807
UNITS: ns
VAR_TYPE: support_data
SCALETYP: linear
MONOTON: INCREASE
TIME_BASE: J2000
TIME_SCALE: Terrestrial Time
REFERENCE_POSITION: Rotating Earth Geoid
<<: *default
CATDESC: Time, number of nanoseconds since J2000 with leap seconds included
FIELDNAM: epoch
LABLAXIS: epoch
FILLVAL: -9223372036854775808
FORMAT: " " # Supposedly not required, fails in xarray_to_cdf
VALIDMIN: -9223372036854775808
VALIDMAX: 9223372036854775807
UNITS: ns
VAR_TYPE: support_data
SCALETYP: linear
MONOTON: INCREASE
TIME_BASE: J2000
TIME_SCALE: Terrestrial Time
REFERENCE_POSITION: Rotating Earth Geoid

# <=== Labels ===>
energy_label:
CATDESC: Energy per charge (E/q) sweeping step
DISPLAY_TYPE: ""
FIELDNAM: Energy step
FILLVAL: ""
FORMAT: A3
LABLAXIS: ""
REFERENCE_POSITION: ""
RESOLUTION: ""
TIME_BASE: ""
TIME_SCALE: ""
UNITS: ""
VALIDMAX: ""
VALIDMIN: ""
VAR_TYPE: metadata

# <=== Dataset Attributes ===>
# TODO: resolve ISTP warning: Width of F10 FORMAT may be insufficient for
# values. Adjust FORMAT or VALIDMIN/VALIDMAX value to resolve.
acquisition_times_attrs:
<<: *default
CATDESC: Time of acquisition for the energy step
DEPEND_1: energy
FIELDNAM: Acquisition Time
FILLVAL: 1.7976931348623157e+308
FILLVAL: .NAN
FORMAT: F10.3
LABLAXIS: Acq Time
UNITS: ms
VALIDMIN: 0
VALIDMAX: 1.7976931348623157e+308
VALIDMAX: 1000
VAR_TYPE: support_data

counters_attrs: &counters
Expand All @@ -71,16 +87,15 @@ counters_attrs: &counters
DEPEND_1: energy
DISPLAY_TYPE: time_series
FIELDNAM: Fill in at creation
FILLVAL: 1.7976931348623157e+308
FORMAT: F10.3
LABL_PTR_1: energy_label
UNITS: rates
UNITS: counts
VALIDMIN: 0
VALIDMAX: 1.7976931348623157e+308
VALIDMAX: 8388607 # max value for a signed 24-bit integer

esa_sweep_attrs:
<<: *default
CATDESC: ElectroStatic Analyzer Energy Values
DEPEND_1: energy
FIELDNAM: ESA Voltage
FORMAT: I19
LABLAXIS: ESA V
Expand Down
2 changes: 2 additions & 0 deletions imap_processing/codice/codice_l0.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ def decom_packets(packet_file: Path) -> list:
"imap_codice_l0_hi-counters-singles_20240429_v001.pkts": "P_COD_HI_INST_COUNTS_SINGLES.xml", # noqa
"imap_codice_l0_hi-omni_20240429_v001.pkts": "P_COD_HI_OMNI_SPECIES_COUNTS.xml",
"imap_codice_l0_hi-sectored_20240429_v001.pkts": "P_COD_HI_SECT_SPECIES_COUNTS.xml", # noqa
"imap_codice_l0_hi-pha_20240429_v001.pkts": "P_COD_HI_PHA.xml",
"imap_codice_l0_hskp_20100101_v001.pkts": "P_COD_NHK.xml",
"imap_codice_l0_lo-counters-aggregated_20240429_v001.pkts": "P_COD_LO_INST_COUNTS_AGGREGATED.xml", # noqa
"imap_codice_l0_lo-counters-singles_20240429_v001.pkts": "P_COD_LO_INST_COUNTS_SINGLES.xml", # noqa
Expand All @@ -47,6 +48,7 @@ def decom_packets(packet_file: Path) -> list:
"imap_codice_l0_lo-nsw-priority_20240429_v001.pkts": "P_COD_LO_NSW_PRIORITY_COUNTS.xml", # noqa
"imap_codice_l0_lo-sw-species_20240429_v001.pkts": "P_COD_LO_SW_SPECIES_COUNTS.xml", # noqa
"imap_codice_l0_lo-nsw-species_20240429_v001.pkts": "P_COD_LO_NSW_SPECIES_COUNTS.xml", # noqa
"imap_codice_l0_lo-pha_20240429_v001.pkts": "P_COD_LO_PHA.xml",
}

xtce_document = Path(
Expand Down
Loading
Loading