Sliceable MetaData Class

changelog/455.feature.rst

@@ -0,0 +1 @@
+Add Sliceable `~ndcube.meta.Meta` class for axis-associated metadata.

docs/explaining_ndcube/metadata.rst

@@ -0,0 +1,291 @@
+.. _ndmeta:
+
+*****************
+Handling Metadata
+*****************
+
+`ndcube`'s data objects do not enforce any requirements on the object assigned to their ``.meta`` attributes.
+However, it does provide an optional object for handling metadata, `~ndcube.NDMeta`.
+This class inherits from `dict`, but provides additional functionalities, chief among which is the ability to associate metadata with data array axes.
+This enables `~ndcube.NDMeta` to update itself through certain operations, e.g. slicing, so that the metadata remains consistent with the associated ND object.
+In this section, we explain the needs that `~ndcube.NDMeta` serves, the concepts underpinning it, and the functionalities it provides.
+
+.. _meta_concepts:
+
+Key Concepts
+============
+
+.. _coords_vs_meta:
+
+Coordinates vs. Axis-aware Metadata
+-----------------------------------
+
+The difference between coordinates and axis-aware metadata is a subtle but important one.
+Formally, a coordinate is a physical space sampled by one or more data axis, whereas axis-aware metadata is information describing the data that can alter along one or more physical dimension.
+An informative example is the difference between time and exposure time.
+The temporal axis of a 3-D image cube samples the physical dimension of time in a strictly increasing monotonic way.
+The times along the temporal axis are therefore coordinate values, not metadata.
+Additionally, a scalar timestamp of a 2-D image is also considered a coordinate in the `ndcube` framework.
+This is because it describes where in the physical dimension of time the data has been sampled, even though it does not correspond to a data axis.
+Because of this, such scalar coordinates are stored in `~ndcube.GlobalCoords`, while coordinates associated with array/pixel axes are stored in the WCS or `~ndcube.ExtraCoords`.
+(See the :ref:`global_coords` section for more discussion on the difference between global and other coordinates.)
+
+By contrast, exposure time describes the interval over which each image was accumulated.
+Exposure time can remain constant, increase or decrease with time, and may switch between these regimes during the time extent of the image cube.
+Like a coordinate, it should be associated with the image cube's temporal axis.
+However, exposure time is reflective of the telescope's operational mode, not a sampling of a physical dimension.
+Exposure time is therefore metadata, not a coordinate.
+
+One reason why it is important to distinguish between coordinates and axis-aware metadata is `ndcube`'s dependence on WCS.
+Most WCS implementations require that there be a unique invertible mapping between pixel and world coordinates, i.e., there is only one pixel value that corresponds to a specific real world value (or combination of such if the coordinate is multi-dimensionsal), and vice versa.
+Therefore, while there may be exceptions for rare and exotic WCS implementations, a good rule of thumb for deciding whether something is a coordinate is:
+coordinates are numeric and strictly monotonic.  Otherwise you have metadata.
+
+The keen-eyed reader may have realised of the above framework that, while not all axis-aligned metadata can be treated as coordinates, all coordinates can be treated like axis-aware metadata.
+This raises the question of why not dispense with coordinates altogether and only have axis-aligned metadata?
+The reason is that the stricter requirements on coordinates have led to a host of powerful coordinate infrastructures that are not valid for generalised axis-aware metadata.
+These include functional WCS implementations which save memory as well as saving compute time through operations such as interpolation, and `~astropy.visualization.wcsaxes.WCSAxes`, which make complicated coordinate-aware plotting easy.
+Therefore, where appropriate, it is beneficial to store coordinates separately from axis-aware metadata.
+
+.. _axis_and_grid_aligned_metadata:
+
+Types of Axis-aware Metadata: Axis-aligned vs. Grid-aligned
+-----------------------------------------------------------
+
+There are two types of axis-aware metadata: axis-aligned and grid-aligned.
+Axis-aligned metadata associates a scalar or string with an array axis.
+It can also assign an array of scalars or strings to multiple array axes, so long as there is one value per associated axis.
+For example, the data produced by a scanning slit spectrograph is associated with real world values.
+But each axis also corresponds to features of the instrument: dispersion (spectral), pixels along the slit (spatial), position of the slit in the rastering sequence (spatial and short timescales), and the raster number (longer timescales).
+The axis-aligned metadata concept allows us to avoid ambiguity by assigning each axis with a label (e.g. ``("dispersion", "slit", "slit step", "raster")``).
+
+By contrast, grid aligned metadata assigns a value to each pixel along axes.
+The exposure time discussion above is an example of 1-D grid-aligned metadata.
+However, grid-aligned metadata can also be multi-dimensional.
+For example, a pixel-dependent response function could be represented as grid-aligned metadata associated with 2 spatial axes.
+
+`~ndcube.NDMeta` supports both axis-aligned and grid-aligned metadata with the same API, which will be discussed in the next section.
+
+.. _ndmeta:
+
+
+NDMeta
+======
+
+.. _initializing_ndmeta:
+
+Initializing an NDMeta
+----------------------
+
+To initialize an `~ndcube.NDMeta`, simply provide it with a `~collections.abc.Mapping` object, e.g. a `dict` or `astropy.io.fits.header.Header`.
+
+.. code-block:: python
+
+  >>> from ndcube import NDMeta
+  >>> raw_meta = {"salutation": "hello", "name": "world"}
+  >>> meta = NDMeta(raw_meta)
+
+We can now access each piece of metadata by indexing ``meta`` as if it were a `dict`:
+
+.. code-block:: python
+
+  >>> meta["name"]
+  'world'
+
+In this example we have provided a very simple set of metadata.
+In fact, it is so simple that there is no practical difference between ``meta`` and a simple `dict`.
+To demonstrate one of the additional functionalities of `~ndcube.NDMeta`, let reinstantiate ``meta``, adding some comments to the metadata.
+To do this, we provide another `~collections.abc.Mapping`, e.g. a `dict`, with the same keys as the main metadata keys, or a subset of them, to the ``key_comments`` kwarg.
+
+.. code-block:: python
+
+
+  >>> key_comments = {"name": "Each planet in the solar system has a name."}
+  >>> meta = NDMeta(raw_meta, key_comments=key_comments)
+
+We can now access the comments by indexing the `~ndcube.NDMeta.key_comments` property:
+
+.. code-block:: python
+
+  >>> meta.key_comments["name"]
+  'Each planet in the solar system has a name.'
+
+Now let's discuss how to initialize how to `~ndcube.NDMeta` with axis-aware metadata.
+(Here, we will specifically consider grid-aligned metadata.  Axis-aligned metadata is assigned in the same way.  But see the :ref:`assigning_axis_aligned_metadata` section for more details.)
+Similar to ``key_comments``, we assign metadata to axes by providing a `~collections.abc.Mapping`, e.g. a `dict`, via its ``axes`` kwarg.
+And like with ``key_comments``, the keys of ``axes`` must be the same, or a subset of, the main metadata keys.
+The axis value must be an `int` or `tuple` of `int` giving the array axes of the data that correspond to the axes of the metadata.
+Note that this means that metadata can be multidimensional.
+Let's say we want to add exposure time that varies with the 1st (temporal) axis of that data, and a pixel response that varies with time and pixel column (1st and 3rd axes).
+
+.. code-block:: python
+
+  >>> import astropy.units as u
+  >>> import numpy as np
+  >>> raw_meta["exposure time"] = [1.9, 2.1, 5, 2, 2] * u.s
+  >>> raw_meta["pixel response"] = np.array([[100., 100., 100., 90., 100.], [85., 100., 90., 100., 100.]]) * u.percent
+  >>> axes = {"exposure time": 0, "pixel response": (0, 2)}
+  >>> meta = NDMeta(raw_meta, axes=axes)
+
+It is easy to see which axes a piece of metadata corresponds to by indexing the `~ndcube.NDMeta.axes` property:
+
+.. code-block:: python
+
+  >>> meta.axes["exposure time"]
+  array([0])
+  >>> meta.axes["pixel response"]
+  array([0, 2])
+
+Finally, it is possible to attach the shape of the associated data to the `~ndcube.NDMeta` instance via the ``data_shape`` kwarg:
+
+.. code-block:: python
+
+  >>> meta = NDMeta(raw_meta, axes=axes, key_comments=key_comments, data_shape=(5, 1, 2))
+
+Or by directly setting the ``~ndcube.NDMeta.data_shape`` property after instantiation:
+
+.. code-block:: python
+
+  >>> meta = NDMeta(raw_meta, axes=axes, key_comments=key_comments)
+  >>> meta.data_shape = (5, 1, 2)
+
+Note that the ``data_shape`` must be compatible with the shapes and associated axes of any axis-aware metadata.
+For example, we couldn't set the length of the first axis to ``6``, because ``meta["exposure time"]`` is associated with the first axis and has a length of ``5``.
+If no ``data_shape`` is provided, it is determined from the axis-aware metadata, if any is provided.
+See the :ref:`data_shape` section for more details.
+
+.. _adding_removing_metadata:
+
+Adding and Removing Metadata
+----------------------------
+
+Because `~ndcube.NDMeta` is a subclass of `dict`, it is possible to add new metadata via the simple ``__setitem__`` API, e.g ``meta[new_key] = new_value``.
+However, this API is not sufficient if we want to add axis-aware or commented metadata.
+This is why `~ndcube.NDMeta` provides an `~ndcube.NDMeta.add` method.
+This method requires the key and value of the new metadata, an optionally accepts a comment and/or axes.
+Let's use this method to add a voltage that varies with time, i.e. the first data axis.
+
+.. code-block:: python
+
+  >>> meta.add("voltage", u.Quantity([1.]*5, unit=u.V), key_comment="detector bias voltage can vary with time and pixel column.", axes=(0,))
+  >>> meta["voltage"]
+  <Quantity [1., 1., 1., 1., 1.] V>
+
+If you try to add metadata with a pre-existing key, `~ndcube.NDMeta.add` will error.
+To replace the value, comment, or axes values of pre-existing metadata, set the ``overwrite`` kwarg to ``True``.
+
+.. code-block:: python
+
+  >>> meta.add("voltage", u.Quantity([-300.]*5, unit=u.V), comment="detector bias voltage", axes=(0,), overwrite=True)
+  >>> meta["voltage"]
+  <Quantity [-300., -300., -300., -300., -300.] V>
+
+Unwanted metadata can be removing by employing the `del` operator.
+
+.. code-block:: python
+
+  >>> del meta["voltage"]
+  >>> meta.get("voltage", "deleted")
+  'deleted'
+
+Note that the `del` operator also removes associated comments and axes.
+
+.. code-block:: python
+
+  >>> del meta["voltage"]
+  >>> meta.key_comments.get("voltage", "deleted")
+  'deleted'
+  >>> meta.axes.get("voltage", "deleted")
+  'deleted'
+
+Data Shape
+----------
+
+The `~ndcube.NDMeta.data_shape` property tracks the shape of the data with which the metadata is associated.
+We have already seen in the :ref:`initialzing_ndmeta` section, that it can be assigned during initialization or by subsequently setting the `~ndcube.NDMeta.data_shape` property directly.
+However, if the ``data_shape`` is not provided, it is inferred from the shapes of axis-aware metadata.
+If no axis-aware metadata is present, `~ndcube.NDMeta.data_shape` is empty:
+
+.. code-block:: python
+
+  >>> from ndcube import NDMeta
+  >>> raw_meta = {"salutation": "hello", "name": "world"}
+  >>> meta = NDMeta(raw_meta)
+  >>> meta.data_shape
+  array([], dtype=int64)
+
+If we now add the ``"pixel response"`` metadata that we used, earlier the `~ndcube.NDMeta.data_shape` will be updated.
+
+.. code-block:: python
+
+  >>> meta.add("pixel response", np.array([[100., 100., 100., 90., 100.], [85., 100., 90., 100., 100.]]) * u.percent, axes=(0, 2))
+  >>> meta.data_shape
+  array([5, 0, 2])
+
+Note that since ``"pixel response"`` is associated with the 1st and 3rd axes, those axes now have the same shape as ``"pixel response"``.
+The existence of a 3rd axis, implies the presence of a 2nd.
+However, we have no metadata associated with it, and hence no knowledge of its length.
+It has therefore been assigned a length of ``0``.
+
+Now that the shape has been set for the 1st and 3rd axes, subsequently added grid-aligned metadata associated with those axes must be compatible with those axis lengths.
+For example, if we add a 1-D ``"exposure time"`` and associate it with the 1st axis, it must have a length of of ``5``, otherwise an error will be raised:
+
+.. code-block:: python
+
+  >>> meta.add("exposure time", [1.9, 2.1, 5, 2, 2] * u.s, axes=0)
+
+Moreover, if we now directly set the `~ndcube.NDMeta.data_shape` via ``meta.data_shape = new_shape``, we cannot change the length of axes already associated with grid-aligned metadata, without first removing or altering that metadata.
+However, these restrictions do not apply if we want to change the shape of the 2nd axis, or add new metadata to it, because its length is ``0``, and hence considered undefined.
+
+.. code-block:: python
+
+  >>> meta.add("row temperature", [-10, -11, -12] * u.deg_C, axes=1)
+  >>> meta.data_shape
+  array([5, 3, 2])
+
+.. _assigning_axis_aligned_metadata
+
+Assigning Axis-aligned Metadata
+-------------------------------
+
+So far, we have only dealt with grid-aligned metadata, i.e. axis-aware metadata which provides a value for each pixel.
+To provide axis-aligned metadata, i.e. where each axis has a single value (see :ref:`axis_and_grid_aligned_metadata`), provide a scalar or string for a single axis, or a 1-D array-like with the same length as the number of associated axes for multi-axis-aligned metadata.
+
+.. code-block:: python
+
+  >>> meta.add("axis name", np.array(["a", "b", "c", "d"]), axes=(0, 1, 2, 3))
+
+Note that the length of ``"axis name"`` is the same as the number of its associated axes.
+Also note that we have now indicated that there is 4th axis.
+``meta.data_shape`` has therefore been automatically updated accordingly.
+
+.. code-block:: python
+
+  >>> meta.data_shape
+  array([5, 3, 2, 0])
+
+However, because axis-aligned metadata does not tell us about the length of the axes, the 4th axis has been assigned a length of zero.
+
+Original_meta
+-------------
+
+As metadata is added, removed, and altered through certain operations, it may still be desirable to refer back to the initial state of the metadata.
+This is the purpose of the `ndcube.NDMeta.original_meta` property.
+It stores the metadata that was originally passed to the `~ndcube.NDMeta` constructor, and it never altered.
+
+.. code-block:: python
+
+  >>> raw_meta = {"salutation": "hello", "name": "world"}
+  >>> meta = NDMeta(raw_meta)
+  >>> del meta["name"]
+  >>> meta.add("exclamation", "!")
+  >>> meta
+  ???
+  >>> meta.original_meta
+
+Note that, ``meta.original_meta`` does not contain ``"exclamation"``, but still contains ``"name"``.
+This is because these were added and removed after initialization.
+Also note that the type of the original metadata object is maintained.
+
+The `~ndcube.NDMeta.original_shape` property is a useful reference back to the original metadata, even after it has been altered via a complex sequence of operations.
+In the :ref:`meta_slicing` section, we discuss the most common of these, slicing.