diff --git a/docs/source/code-documentation/tools/xtce-generator.rst b/docs/source/code-documentation/tools/xtce-generator.rst index 452746a8d..cc16f5066 100644 --- a/docs/source/code-documentation/tools/xtce-generator.rst +++ b/docs/source/code-documentation/tools/xtce-generator.rst @@ -1,84 +1,166 @@ .. _xtce_generator: -Generating Telemetry XML with Python Script -=========================================== +XML Telemetric and Command Exchange (XTCE) +========================================== -Here is some info on `XTCE `_. This Green -Book introduces the main concepts of XML Telemetric and Command Exchange (XTCE), a -telemetry and telecommand database format for spacecraft monitoring -and control. +The `XTCE green book `_. +introduces the main concepts and specifications of XTCE. +The XTCE format is a specification for spacecraft monitoring and control data transfer. +It can be used to define how packets can be sent and received from the spacecraft, +which we then unpack on the ground using the XTCE format. -General -------- - -This document provides steps and information on how to use -`xtce_generator_template.py` script as a base for users to generate -telemetry XML files. The script is designed to simplify the process of creating -telemetry definitions for various packet types. - -The script is located in the `tools/xtce_generation` directory. The script is called -`xtce_generator_template.py`. The script is a ``template`` that can be modified to -generate telemetry XML files for different packet types. Your new file should be -called `xtce_generator_yourinstrument.py`. -An example of how to use the script is `xtce_generator_codice.py` which is also -located in the `tools/xtce_generation` directory. Before you Start ---------------- Generating XTCEs is only done whenever packet definitions get updated, and thus it -is not a part of the main processing package. To use it there are a few extra -dependencies like ``pandas`` that you can install with +is not regularly run as a part of processing. To use it there are a few extra +dependencies (like ``openpyxl`` for reading excel spreadsheets) that you +can install with the tools extra. .. code:: + # with poetry poetry install --extras tools + # or with pip + pip install imap_processing[tools] How to Use ---------- -Define the instrument name in the `main()` function by setting the `instrument_name` -variable to the name of your instrument. - -.. code:: - - instrument_name = "your_instrument_name" - -In the code, file paths are being configured. Make sure to change the file paths to -match your instrument's file structure. - -.. code:: - - current_directory = Path(__file__).parent - module_path = f"{current_directory}/../../imap_processing" - # This is the path of the output directory - packet_definition_path = f"{module_path}/{instrument_name}/packet_definitions" - # This is the path to the excel file that contains the telemetry definitions - path_to_excel_file = f"{current_directory}/your_packet.xlsx" - -Define packet names and `Application Process Identifiers (APIDs) -`_. -The packet names are **case sensitive** meaning the the packet names need to be exactly -what the tabs of the spreadsheet are. APID's must match the names and apIds in the -packet definition file. You can use as many packet names and apIds as you want. -The APID should be an integer (not hexadecimal). -Follow the format below. - -.. code:: - - packets = { - # Define packet names and associated Application IDs (apId) - "your_packet_A": ####, - "your_packet_B": ####, - # ... (other packet definitions) - } - -Generating Telemetry XML Files -------------------------------- - -Once you have your xtce processing file defined, you can run it with the -following command: +There is a command line utility ``imap_xtce`` that can be used to generate XTCE files +that is installed with the ``imap_processing`` package. +The utility takes in an excel file and generates XTCE files for each packet definition +in the excel file. If you don't provide an output file, it will generate the XTCE file +with the same name as the input Excel file but with the extension changed to ``.xml``. .. code:: - python xtce_generator_instrument_name.py + imap_xtce path/to/excel_packet_file.xlsx --output path/to/output_packet_definition.xml + + +Spreadsheet definitions +----------------------- + +The XTCE generator uses an excel spreadsheet to define the packet structure. +This is a commonly used spreadsheet format at the Laboratory for Atmospheric and Space Physics (LASP). +The required tabs are ``Subsystem``, ``Packets``, and whatever packet names you have. + +Subsystem tab +~~~~~~~~~~~~~ + +The ``Subsystem`` tab is used to define the instrument name and last updated date of the packet data. + +.. list-table:: Subsystem + :header-rows: 1 + + * - infoField + - infoValue + * - subsystem + - MY_INSTRUMENT + * - sheetReleaseDate + - 01/01/2010 + * - sheetReleaseRev + - 1.2.3 + +Packets tab +~~~~~~~~~~~ + +The packets tab contains the list of packets that you want to include within your XTCE +packet definition. You can remove rows from this to control which individual packet tabs +are read in later. The ``packetName`` column defines which other tabs to read in. So in +the following table, the generator will read in the ``MY_INSTRUMENT_HK`` and +``MY_INSTRUMENT_SCI`` tabs that contain the packet definitions. + +.. note:: + The generator will also work with tabs prefixed with ``P_``, so ``P_MY_INSTRUMENT_HK`` and + ``P_MY_INSTRUMENT_SCI`` tab names would also work. + +.. list-table:: Packets + :header-rows: 1 + + * - packetName + - apId + * - MY_INSTRUMENT_HK + - 123 + * - MY_INSTRUMENT_SCI + - 124 + +Individual packet tabs +~~~~~~~~~~~~~~~~~~~~~~ + +Each packet tab contains the contents that will create the XTCE packet definition. +The required columns are ``packetName``, ``mnemonic``, ``lengthInBits``, ``dataType``, +``convertAs``, with optional ``shortDescription`` and ``longDescription`` columns. + +Within the XTCE definition, the variable names will be ``packetName.mnemonic`` separated +with a period for easier distinguishing between packets and variables. For example, +the table below would have this XTCE parameter definition ``MY_INSTRUMENT_HK.VARIABLE1_UINT`` +for the first variable. If an analog conversion is required, the ``convertAs`` column +should be set to ``ANALOG``, which will then look at the ``AnalogConversions`` tab for +the conversion details. + +.. list-table:: MY_INSTRUMENT_HK + :header-rows: 1 + + * - packetName + - mnemonic + - lengthInBits + - dataType + - convertAs + - shortDescription + - longDescription + * - MY_INSTRUMENT_HK + - VARIABLE1_UINT + - 3 + - UINT + - NONE + - My short variable description + - My verbose variable description + * - MY_INSTRUMENT_HK + - VARIABLE2_CONVERTED + - 3 + - UINT + - ANALOG + - Apply an analog conversion + - + * - MY_INSTRUMENT_HK + - VARIABLE_LENGTH_BINARY_SCIENCE + - 100 + - BYTE + - NONE + - + - This variable size will be dynamic and based on the packet size + +AnalogConversions tab (optional) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Packet parsing can also apply analog conversions to the data being read in. +For example, to change from a raw unsigned integer value to a temperature in Kelvin. +The ``AnalogConversions`` tab is used to define these conversions. +It currently only supports unsegmented polynomial conversions, and looks for the +coefficients defined from ``c0`` to ``c7`` to define the order of the polynomial. + +.. list-table:: AnalogConversions + :header-rows: 1 + + * - packetName + - mnemonic + - c0 + - c1 + - c2 + - c3 + - c4 + - c5 + - c6 + - c7 + * - MY_INSTRUMENT_HK + - VARIABLE2_CONVERTED + - 123.456 + - 0.234 + - + - + - + - + - + - diff --git a/imap_processing/cdf/config/imap_hi_variable_attrs.yaml b/imap_processing/cdf/config/imap_hi_variable_attrs.yaml index a450acc34..f71c96e57 100644 --- a/imap_processing/cdf/config/imap_hi_variable_attrs.yaml +++ b/imap_processing/cdf/config/imap_hi_variable_attrs.yaml @@ -287,7 +287,6 @@ hi_de_nominal_bin: <<: *default_uint8 CATDESC: Corresponding histogram angle bin for this Direct Event FIELDNAM: Histogram Bin Number - FILLVAL: 511 FORMAT: I2 LABLAXIS: Hist Bin \# VALIDMIN: 0 diff --git a/imap_processing/mag/l1a/mag_l1a_data.py b/imap_processing/mag/l1a/mag_l1a_data.py index 58130b9d6..b808d66c7 100644 --- a/imap_processing/mag/l1a/mag_l1a_data.py +++ b/imap_processing/mag/l1a/mag_l1a_data.py @@ -336,6 +336,10 @@ def to_signed16(n: int) -> int: primary_vectors = [] secondary_vectors = [] + # To avoid overflows, we need to cast the potentially 8 bit signed integers to + # int32 before the bitshifting operations below. + vector_data = vector_data.astype(np.int32) + # Since the vectors are stored as 50 bit chunks but accessed via hex (4 bit # chunks) there is some shifting required for processing the bytes. # However, from a bit processing perspective, the first 48 bits of each 50 bit diff --git a/imap_processing/spice/kernels.py b/imap_processing/spice/kernels.py index 1ff30df21..2d4e21154 100644 --- a/imap_processing/spice/kernels.py +++ b/imap_processing/spice/kernels.py @@ -1 +1,158 @@ """Functions for furnishing and tracking SPICE kernels.""" + +import functools +import logging +import os +from typing import Any, Callable, Optional + +import spiceypy as spice +from spiceypy.utils.exceptions import SpiceyError + +logger = logging.getLogger(__name__) + + +def ensure_spice( + f_py: Optional[Callable] = None, time_kernels_only: bool = False +) -> Callable: + """ + Decorator/wrapper that automatically furnishes SPICE kernels. + + Parameters + ---------- + f_py : Callable + The function requiring SPICE that we are going to wrap if being used + explicitly, otherwise None, in which case ensure_spice is being used, + not as a function wrapper (see l2a_processing.py) but as a true + decorator without an explicit function argument. + time_kernels_only : bool + Specify that we only need to furnish time kernels (if SPICE_METAKERNEL + is set, we still just furnish that metakernel and assume the time + kernels are included. + + Returns + ------- + Callable + Decorated function, with spice error handling. + + Notes + ----- + Before trying to understand this piece of code, read this: + https://stackoverflow.com/questions/5929107/decorators-with-parameters/60832711#60832711 + + **Control flow overview:** + 1. Try simply calling the wrapped function naively. + * SUCCESS? Great! We're done. + * SpiceyError? Go to step 2. + + 2. Furnish metakernel at SPICE_METAKERNEL + * SUCCESS? Great, return the original function again (so it can be + re-run). + * KeyError? Seems like SPICE_METAKERNEL isn't set, no problem. Go to + step 3. + + 3. Did we get the parameter time_kernels_only=True? + --> YES? We only need LSK and SCLK kernels to run this function. Go fetch + those and furnish and return the original function (so it can be re-run). + --> NO? Dang. This is sort of the end of the line. Re-raise the error + generated from the failed spiceypy function call but add a better + message to it. + + Examples + -------- + There are three ways to use this object + + 1. A decorator with no arguments + >>> @ensure_spice + ... def my_spicey_func(a, b): + ... pass + + 2. A decorator with parameters. This is useful + if we only need the latest SCLK and LSK kernels for the function involved. + >>> @ensure_spice(time_kernels_only=True) + ... def my_spicey_time_func(a, b): + ... pass + + 3. An explicit wrapper function, providing a dynamically set value for + parameters, e.g. time_kernels_only + >>> wrapped = ensure_spice(spicey_func, time_kernels_only=True) + ... result = wrapped(*args, **kwargs) + """ + if f_py and not callable(f_py): + raise ValueError( + f"Received a non-callable object {f_py} as the f_py argument to" + f"ensure_spice. f_py must be a callable object." + ) + + def _decorator(func: Callable[..., Callable]) -> Callable: + """ + Decorate or wrap input function depending on how ensure_spice is used. + + Parameters + ---------- + func : Callable + The function to be decorated/wrapped. + + Returns + ------- + Callable + If used as a function wrapper, the decorated function is returned. + """ + + @functools.wraps(func) + def wrapper_ensure_spice(*args: Any, **kwargs: Any) -> Any: + """ + Wrap the function that ensure_spice is used on. + + Parameters + ---------- + *args : list + The positional arguments passed to the decorated function. + **kwargs + The keyword arguments passed to the decorated function. + + Returns + ------- + Object + Output from wrapped function. + """ + try: + # Step 1. + return func( + *args, **kwargs + ) # Naive first try. Maybe SPICE is already furnished. + except SpiceyError as spicey_err: + try: + # Step 2. + metakernel_path = os.environ["SPICE_METAKERNEL"] + spice.furnsh(metakernel_path) + except KeyError: + # TODO: An additional step that was used on EMUS was to get + # a custom metakernel from the SDC API based on an input + # time range. + if time_kernels_only: + # Step 3. + # TODO: Decide if this is useful for IMAP. Possible + # implementation could include downloading + # the most recent leapsecond kernel from NAIF (see: + # https://lasp.colorado.edu/nucleus/projects/LIBSDC/repos/libera_utils/browse/libera_utils/spice_utils.py + # for LIBERA implementation of downloading and caching + # kernels) and finding the most recent IMAP clock + # kernel in EFS. + raise NotImplementedError from spicey_err + else: + raise SpiceyError( + "When calling a function requiring SPICE, we failed" + "to load a metakernel. SPICE_METAKERNEL is not set," + "and time_kernels_only is not set to True" + ) from spicey_err + return func(*args, **kwargs) + + return wrapper_ensure_spice + + # Note: This return was originally implemented as a ternary operator, but + # this caused mypy to fail due to this bug: + # https://github.com/python/mypy/issues/4134 + if callable(f_py): + return _decorator(f_py) + else: + return _decorator diff --git a/imap_processing/tests/conftest.py b/imap_processing/tests/conftest.py index 4a7c9bd76..b9adc63e3 100644 --- a/imap_processing/tests/conftest.py +++ b/imap_processing/tests/conftest.py @@ -1,7 +1,11 @@ """Global pytest configuration for the package.""" +import re + import imap_data_access +import numpy as np import pytest +import spiceypy as spice from imap_processing import imap_module_directory @@ -18,3 +22,106 @@ def _set_global_config(monkeypatch, tmp_path): @pytest.fixture(scope="session") def imap_tests_path(): return imap_module_directory / "tests" + + +# Furnishing fixtures for testing kernels +# --------------------------------------- +@pytest.fixture(autouse=True) +def _autoclear_spice(): + """Automatically clears out all SPICE remnants after every single test to + prevent the kernel pool from interfering with future tests. Option autouse + ensures this is run after every test.""" + yield + spice.kclear() + + +@pytest.fixture(scope="session") +def spice_test_data_path(imap_tests_path): + return imap_tests_path / "spice/test_data" + + +@pytest.fixture() +def furnish_test_lsk(spice_test_data_path): + """Furnishes (temporarily) the testing LSK""" + test_lsk = spice_test_data_path / "naif0012.tls" + spice.furnsh(test_lsk) + yield test_lsk + spice.kclear() + + +@pytest.fixture() +def furnish_sclk(spice_test_data_path): + """Furnishes (temporarily) the SCLK for JPSS stored in the package data directory""" + test_sclk = spice_test_data_path / "imap_sclk_0000.tsc" + spice.furnsh(test_sclk) + yield test_sclk + spice.kclear() + + +@pytest.fixture() +def use_test_metakernel(monkeypatch, spice_test_data_path): + """For the whole test session, set the SPICE_METAKERNEL environment variable + Prime the test metakernel by creating it from the template metakernel + (allows using absolute paths on any dev system)""" + + def make_metakernel_from_kernels(metakernel, kernels): + """Helper function that writes a test metakernel from a list of filenames""" + with open(metakernel, "w") as mk: + mk.writelines( + [ + "\n", + "\\begintext\n", + "\n", + "This is a temporary metakernel for imap_processing" + " unit and integration testing.\n", + "\n", + "\\begindata\n", + "\n", + "KERNELS_TO_LOAD = (\n", + ] + ) + # Put single quotes around every kernel name + kernels_with_quotes = [" '" + kern + "'" for kern in kernels] + # Add a comma and EOL to the end of each kernel path except the last. + formated_kernels = [kern + ",\n" for kern in kernels_with_quotes[0:-1]] + # Add ')' to the last kernel + formated_kernels.append(kernels_with_quotes[-1] + "\n)\n\n") + mk.writelines(formated_kernels) + + def get_test_kernels_to_load(): + """ + Helper function for grabbing a list of kernel filenames from the test + metakernel template. This is necessary in order to get absolute paths on + any system. Formats the absolute paths using the test data path fixture + value. + """ + test_metakernel = spice_test_data_path / "imap_test_metakernel.template" + kernels_to_load = [] + max_line_length = 80 + with open(test_metakernel) as mk: + for k in mk: + kernel = k.rstrip("\n").format( + **{"SPICE_TEST_DATA_PATH": str(spice_test_data_path.absolute())} + ) + while len(kernel) > 0: + if len(kernel) <= max_line_length: + kernels_to_load.append(kernel) + break + else: + slash_positions = np.array( + [m.start() for m in re.finditer("/", kernel)] + ) + stop_idx = ( + slash_positions[slash_positions < max_line_length - 1].max() + + 1 + ) + kernels_to_load.append(kernel[0:stop_idx] + "+") + kernel = kernel[stop_idx:] + return kernels_to_load + + metakernel_path = imap_data_access.config["DATA_DIR"] / "imap_2024_v001.tm" + kernels_to_load = get_test_kernels_to_load() + make_metakernel_from_kernels(metakernel_path, kernels_to_load) + monkeypatch.setenv("SPICE_METAKERNEL", str(metakernel_path)) + yield str(metakernel_path) + spice.kclear() diff --git a/tools/tests/test_data/spice/imap_sclk_0000.tsc b/imap_processing/tests/spice/test_data/imap_sclk_0000.tsc similarity index 100% rename from tools/tests/test_data/spice/imap_sclk_0000.tsc rename to imap_processing/tests/spice/test_data/imap_sclk_0000.tsc diff --git a/imap_processing/tests/spice/test_data/imap_test_metakernel.template b/imap_processing/tests/spice/test_data/imap_test_metakernel.template new file mode 100644 index 000000000..b4a5d9f82 --- /dev/null +++ b/imap_processing/tests/spice/test_data/imap_test_metakernel.template @@ -0,0 +1,2 @@ +{SPICE_TEST_DATA_PATH}/imap_sclk_0000.tsc +{SPICE_TEST_DATA_PATH}/naif0012.tls \ No newline at end of file diff --git a/tools/tests/test_data/spice/naif0012.tls b/imap_processing/tests/spice/test_data/naif0012.tls similarity index 100% rename from tools/tests/test_data/spice/naif0012.tls rename to imap_processing/tests/spice/test_data/naif0012.tls diff --git a/imap_processing/tests/spice/test_kernels.py b/imap_processing/tests/spice/test_kernels.py index 059d0f231..c8ce926ad 100644 --- a/imap_processing/tests/spice/test_kernels.py +++ b/imap_processing/tests/spice/test_kernels.py @@ -1 +1,69 @@ """Tests coverage for imap_processing/spice/kernels.py""" + +import pytest +import spiceypy as spice +from spiceypy.utils.exceptions import SpiceyError + +from imap_processing.spice import kernels + + +@kernels.ensure_spice +def single_wrap_et2utc(et, fmt, prec): + """Directly decorate a spice function with ensure_spice for use in tests""" + return spice.et2utc(et, fmt, prec) + + +@kernels.ensure_spice +def double_wrap_et2utc(et, fmt, prec): + """Decorate a spice function twice with ensure_spice for use in tests. This + simulates some decorated outer functions that call lower level functions + that are already decorated.""" + return single_wrap_et2utc(et, fmt, prec) + + +@kernels.ensure_spice(time_kernels_only=True) +def single_wrap_et2utc_tk_only(et, fmt, prec): + """Directly wrap a spice function with optional time_kernels_only set True""" + return spice.et2utc(et, fmt, prec) + + +@kernels.ensure_spice(time_kernels_only=True) +def double_wrap_et2utc_tk_only(et, fmt, prec): + """Decorate a spice function twice with ensure_spice for use in tests. This + simulates some decorated outer functions that call lower level functions + that are already decorated.""" + return single_wrap_et2utc(et, fmt, prec) + + +@pytest.mark.parametrize( + "func", + [ + single_wrap_et2utc, + single_wrap_et2utc_tk_only, + double_wrap_et2utc, + double_wrap_et2utc_tk_only, + ], +) +def test_ensure_spice_emus_mk_path(func, use_test_metakernel): + """Test functionality of ensure spice with SPICE_METAKERNEL set""" + assert func(577365941.184, "ISOC", 3) == "2018-04-18T23:24:31.998" + + +def test_ensure_spice_time_kernels(): + """Test functionality of ensure spice with timekernels set""" + wrapped = kernels.ensure_spice(spice.et2utc, time_kernels_only=True) + # TODO: Update/remove this test when a decision has been made about + # whether IMAP will use the time_kernels_only functionality and the + # ensure_spice decorator has been update. + with pytest.raises(NotImplementedError): + _ = wrapped(577365941.184, "ISOC", 3) == "2018-04-18T23:24:31.998" + + +def test_ensure_spice_key_error(): + """Test functionality of ensure spice when all branches fail""" + wrapped = kernels.ensure_spice(spice.et2utc) + # The ensure_spice decorator should raise a SpiceyError when all attempts to + # furnish a set of kernels with sufficient coverage for the spiceypy + # functions that it decorates. + with pytest.raises(SpiceyError): + _ = wrapped(577365941.184, "ISOC", 3) == "2018-04-18T23:24:31.998" diff --git a/poetry.lock b/poetry.lock index 4f8f4bb36..83c5479d1 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1747,4 +1747,4 @@ tools = ["openpyxl", "pandas"] [metadata] lock-version = "2.0" python-versions = ">=3.9,<4" -content-hash = "28cbb877e15c362c80541a8cebd35f12575cd86802c902505b4db1f65640d37e" +content-hash = "f750033025b765826c827adb1142fd59444e13a985755c2a6efc3a207348a959" diff --git a/pyproject.toml b/pyproject.toml index df0ef2a95..598c512e7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -38,7 +38,7 @@ space_packet_parser = ">=4.2.0" spiceypy = ">=6.0.0" xarray = '>=2023.0.0' pyyaml = "^6.0.1" -numpy = "^1.26.4" +numpy = "<=3" # Optional dependencies numpydoc = {version="^1.5.0", optional=true} diff --git a/tools/tests/unit/test_spice_examples.py b/tools/tests/unit/test_spice_examples.py index d998a3008..d47a607cd 100644 --- a/tools/tests/unit/test_spice_examples.py +++ b/tools/tests/unit/test_spice_examples.py @@ -4,6 +4,7 @@ import pytest import spiceypy as spice +from imap_processing import imap_module_directory from tools.spice.spice_examples import ( _get_particle_velocity, build_annotated_events, @@ -29,6 +30,12 @@ def kernels(kernel_directory): kernels = list_files_with_extensions( kernel_directory, [".tsc", ".tls", ".tf", ".bsp", ".ck"] ) + # Some kernels were moved into imap_processing package + kernels.extend( + list_files_with_extensions( + imap_module_directory / "tests/spice/test_data", [".tsc", ".tls"] + ) + ) return kernels