Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DOC: Update XTCE generator documentation #766

Merged
merged 1 commit into from
Aug 23, 2024
Merged

DOC: Update XTCE generator documentation #766

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
210 changes: 146 additions & 64 deletions docs/source/code-documentation/tools/xtce-generator.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,84 +1,166 @@
.. _xtce_generator:

Generating Telemetry XML with Python Script
===========================================
XML Telemetric and Command Exchange (XTCE)
==========================================

Here is some info on `XTCE <https://public.ccsds.org/Pubs/660x2g2.pdf/>`_. This Green
Book introduces the main concepts of XML Telemetric and Command Exchange (XTCE), a
telemetry and telecommand database format for spacecraft monitoring
and control.
The `XTCE green book <https://public.ccsds.org/Pubs/660x2g2.pdf>`_.
introduces the main concepts and specifications of XTCE.
The XTCE format is a specification for spacecraft monitoring and control data transfer.
It can be used to define how packets can be sent and received from the spacecraft,
which we then unpack on the ground using the XTCE format.

General
-------

This document provides steps and information on how to use
`xtce_generator_template.py` script as a base for users to generate
telemetry XML files. The script is designed to simplify the process of creating
telemetry definitions for various packet types.

The script is located in the `tools/xtce_generation` directory. The script is called
`xtce_generator_template.py`. The script is a ``template`` that can be modified to
generate telemetry XML files for different packet types. Your new file should be
called `xtce_generator_yourinstrument.py`.
An example of how to use the script is `xtce_generator_codice.py` which is also
located in the `tools/xtce_generation` directory.

Before you Start
----------------

Generating XTCEs is only done whenever packet definitions get updated, and thus it
is not a part of the main processing package. To use it there are a few extra
dependencies like ``pandas`` that you can install with
is not regularly run as a part of processing. To use it there are a few extra
dependencies (like ``openpyxl`` for reading excel spreadsheets) that you
can install with the tools extra.

.. code::
# with poetry
poetry install --extras tools
# or with pip
pip install imap_processing[tools]
How to Use
----------

Define the instrument name in the `main()` function by setting the `instrument_name`
variable to the name of your instrument.

.. code::
instrument_name = "your_instrument_name"
In the code, file paths are being configured. Make sure to change the file paths to
match your instrument's file structure.

.. code::
current_directory = Path(__file__).parent
module_path = f"{current_directory}/../../imap_processing"
# This is the path of the output directory
packet_definition_path = f"{module_path}/{instrument_name}/packet_definitions"
# This is the path to the excel file that contains the telemetry definitions
path_to_excel_file = f"{current_directory}/your_packet.xlsx"
Define packet names and `Application Process Identifiers (APIDs)
<https://sanaregistry.org/r/space_packet_protocol_application_process_id/>`_.
The packet names are **case sensitive** meaning the the packet names need to be exactly
what the tabs of the spreadsheet are. APID's must match the names and apIds in the
packet definition file. You can use as many packet names and apIds as you want.
The APID should be an integer (not hexadecimal).
Follow the format below.

.. code::
packets = {
# Define packet names and associated Application IDs (apId)
"your_packet_A": ####,
"your_packet_B": ####,
# ... (other packet definitions)
}
Generating Telemetry XML Files
-------------------------------

Once you have your xtce processing file defined, you can run it with the
following command:
There is a command line utility ``imap_xtce`` that can be used to generate XTCE files
that is installed with the ``imap_processing`` package.
The utility takes in an excel file and generates XTCE files for each packet definition
in the excel file. If you don't provide an output file, it will generate the XTCE file
with the same name as the input Excel file but with the extension changed to ``.xml``.

.. code::
python xtce_generator_instrument_name.py
imap_xtce path/to/excel_packet_file.xlsx --output path/to/output_packet_definition.xml
Spreadsheet definitions
-----------------------

The XTCE generator uses an excel spreadsheet to define the packet structure.
This is a commonly used spreadsheet format at the Laboratory for Atmospheric and Space Physics (LASP).
The required tabs are ``Subsystem``, ``Packets``, and whatever packet names you have.

Subsystem tab
~~~~~~~~~~~~~

The ``Subsystem`` tab is used to define the instrument name and last updated date of the packet data.

.. list-table:: Subsystem
:header-rows: 1

* - infoField
- infoValue
* - subsystem
- MY_INSTRUMENT
* - sheetReleaseDate
- 01/01/2010
* - sheetReleaseRev
- 1.2.3

Packets tab
~~~~~~~~~~~

The packets tab contains the list of packets that you want to include within your XTCE
packet definition. You can remove rows from this to control which individual packet tabs
are read in later. The ``packetName`` column defines which other tabs to read in. So in
the following table, the generator will read in the ``MY_INSTRUMENT_HK`` and
``MY_INSTRUMENT_SCI`` tabs that contain the packet definitions.

.. note::
The generator will also work with tabs prefixed with ``P_``, so ``P_MY_INSTRUMENT_HK`` and
``P_MY_INSTRUMENT_SCI`` tab names would also work.

.. list-table:: Packets
:header-rows: 1

* - packetName
- apId
* - MY_INSTRUMENT_HK
- 123
* - MY_INSTRUMENT_SCI
- 124

Individual packet tabs
~~~~~~~~~~~~~~~~~~~~~~

Each packet tab contains the contents that will create the XTCE packet definition.
The required columns are ``packetName``, ``mnemonic``, ``lengthInBits``, ``dataType``,
``convertAs``, with optional ``shortDescription`` and ``longDescription`` columns.

Within the XTCE definition, the variable names will be ``packetName.mnemonic`` separated
with a period for easier distinguishing between packets and variables. For example,
the table below would have this XTCE parameter definition ``MY_INSTRUMENT_HK.VARIABLE1_UINT``
for the first variable. If an analog conversion is required, the ``convertAs`` column
should be set to ``ANALOG``, which will then look at the ``AnalogConversions`` tab for
the conversion details.

.. list-table:: MY_INSTRUMENT_HK
:header-rows: 1

* - packetName
- mnemonic
- lengthInBits
- dataType
- convertAs
- shortDescription
- longDescription
* - MY_INSTRUMENT_HK
- VARIABLE1_UINT
- 3
- UINT
- NONE
- My short variable description
- My verbose variable description
* - MY_INSTRUMENT_HK
- VARIABLE2_CONVERTED
- 3
- UINT
- ANALOG
- Apply an analog conversion
-
* - MY_INSTRUMENT_HK
- VARIABLE_LENGTH_BINARY_SCIENCE
- 100
- BYTE
- NONE
-
- This variable size will be dynamic and based on the packet size

AnalogConversions tab (optional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Packet parsing can also apply analog conversions to the data being read in.
For example, to change from a raw unsigned integer value to a temperature in Kelvin.
The ``AnalogConversions`` tab is used to define these conversions.
It currently only supports unsegmented polynomial conversions, and looks for the
coefficients defined from ``c0`` to ``c7`` to define the order of the polynomial.

.. list-table:: AnalogConversions
:header-rows: 1

* - packetName
- mnemonic
- c0
- c1
- c2
- c3
- c4
- c5
- c6
- c7
* - MY_INSTRUMENT_HK
- VARIABLE2_CONVERTED
- 123.456
- 0.234
-
-
-
-
-
-
Loading