Additional ANTS tutorials

docs/source/tutorial_cli_walkthroughs.rst

@@ -0,0 +1,199 @@
+.. meta::
+   :description lang=en: Worked examples for the ANTS command line tools
+   :keywords: ancil_2anc, ancil_create_shapefile, ancil_fill_n_merge, ancil_general_regrid, ancil_vertical_regrid, tutorial
+   :property=og:locale: en_GB
+
+.. include:: common.txt
+
+.. Created with assitance of MetOffice Enterprise Copilot
+
+Worked examples: the ANTS command line tools
+================================================
+
+:doc:`core_capabilities` introduces the five general purpose command line
+tools that ship with ANTS, and their individual pages (:doc:`ancil_2anc`,
+:doc:`ancil_create_shapefile`, :doc:`ancil_fill_n_merge`,
+:doc:`ancil_general_regrid`, :doc:`ancil_vertical_regrid`) document their full
+set of arguments. This tutorial instead walks through a realistic invocation
+of each tool, so you can see how they fit together in practice. 
+
+All five tools share a common command line interface, provided by
+:class:`ants.command_parse.AntsArgParser`:
+
+.. code-block::
+
+    <tool>.py <SOURCE1> [<SOURCE2> ...] --output <OUTPUT> [--ants-config <CONFIG>]
+
+and are typically launched via the ``ants-launch`` wrapper, as shown below.
+
+ancil_2anc: converting between fileformats
+--------------------------------------------
+
+:doc:`ancil_2anc` loads one or more cubes and writes them out as NetCDF and/or
+an F03 ancillary file, inferring as much metadata as it can from the source.
+It is a straight fileformat translation - see :doc:`tutorial_load_save` for
+the underlying load/save mechanics.
+
+.. code-block:: bash
+
+    ancil_2anc.py land_cover_fraction.nc \
+        --output land_cover_fraction_ancil \
+        --grid-staggering 6 \
+        --ants-config rose-app-run.conf
+
+The ``--grid-staggering`` argument (3 for New Dynamics, 6 for ENDGame) is
+often required, since it usually cannot be inferred from a NetCDF source.
+
+ancil_create_shapefile: producing a validity polygon
+------------------------------------------------------
+
+:doc:`ancil_create_shapefile` turns a JSON file containing a list of
+``[longitude, latitude]`` pairs into a shapefile polygon, which can then be
+used as the ``validity_polygon`` input to ``ancil_fill_n_merge`` (see
+:doc:`tutorial_merge_fill`).
+
+A minimal input JSON, defining a simple box:
+
+.. code-block:: json
+
+    [[-10.0, 40.0], [-10.0, 60.0], [10.0, 60.0], [10.0, 40.0]]
+
+.. code-block:: bash
+
+    ancil_create_shapefile.py validity_region.json validity_region.shp
+
+ancil_fill_n_merge: merging and filling
+------------------------------------------
+
+:doc:`ancil_fill_n_merge` merges a primary and (optional) alternate source and
+fills any remaining missing points - see :doc:`tutorial_merge_fill` for the
+concepts. ``--search-method kdtree`` is recommended for consistency across UM
+and LFRic pipelines:
+
+.. code-block:: bash
+
+    ancil_fill_n_merge.py primary_source.nc alternate_source.nc \
+        --output merged_filled \
+        --target-lsm target_lsm.nc \
+        --polygon validity_region.shp \
+        --search-method kdtree \
+        --ants-config rose-app-run.conf
+
+If only ``primary_source.nc`` is provided (no alternate), only the fill stage
+runs.
+
+ancil_general_regrid: regridding to a target grid
+----------------------------------------------------
+
+:doc:`ancil_general_regrid` regrids a source onto a target grid or target land
+sea mask - see :doc:`tutorial_regridding` for the underlying mechanics:
+
+.. code-block:: bash
+
+    ancil_general_regrid.py source_field.nc \
+        --output regridded_field \
+        --target-grid target_grid_namelist \
+        --ants-config rose-app-run.conf
+
+To regrid directly onto a target land sea mask instead of a plain grid, use
+``--target-lsm`` in place of ``--target-grid``; the result will be made
+consistent with that mask using the same fill algorithms discussed in
+:doc:`tutorial_merge_fill`.
+
+This application has no default regridding scheme the ``--ants-config`` file
+passed above (``rose-app-run.conf``) must include a ``[ants_regridding_horizontal]`` 
+section (and a ``[ants_regridding_vertical]`` section too, if the target also
+varies vertically), for example:
+
+.. code-block::
+
+    [ants_regridding_horizontal]
+    scheme = Linear
+
+See :class:`ants.config.GlobalConfiguration` for the full list of valid
+``scheme`` values. Omitting this section raises an error.
+
+.. note:: ``ancil_vertical_regrid`` is available for vertical only regrid operations
+   or cases where you want to ensure a vertical regrid is carried out before any
+   other processing.
+
+
+ancil_vertical_regrid: regridding onto a new vertical level set
+------------------------------------------------------------------
+
+:doc:`ancil_vertical_regrid` is the vertical-only counterpart to
+``ancil_general_regrid``. Rather than moving data onto a new horizontal grid,
+it interpolates onto a new set of vertical levels (for example a target
+vertical namelist), and requires that the source and target already share
+the same horizontal (latitude/longitude) coordinates:
+
+.. code-block:: bash
+
+    ancil_vertical_regrid.py aerosol_3d.nc \
+        --output aerosol_3d_L70 \
+        --target-grid vertlevs_L70_50t_20s_80km \
+        --ants-config rose-app-run.conf
+
+This application has no default vertical scheme - the ``--ants-config`` file
+passed above (``rose-app-run.conf``) must include a
+``[ants_regridding_vertical]`` section specifying one, for example:
+
+.. code-block::
+
+    [ants_regridding_vertical]
+    scheme = Linear
+
+See :class:`ants.config.GlobalConfiguration` for the full list of valid
+``scheme`` values. Omitting this section raises an error.
+
+.. note::
+   ``ancil_vertical_regrid`` checks its target before regridding: if the
+   target is a UGrid mesh, it raises an error directing you to the
+   regrid-to-mesh application in UG-ANTS instead; if the target's
+   latitude/longitude coordinates do not match the source's, it raises an
+   error rather than silently regridding horizontally too. Use
+   :doc:`ancil_general_regrid` (see :doc:`tutorial_regridding`) when a
+   horizontal regrid is also needed.
+
+Decomposition configuration
+------------------------------
+
+All five tools accept ``--ants-config`` to point at an ANTS configuration
+file. This is most commonly used to enable :doc:`decomposition` for large
+datasets, for example:
+
+.. code-block::
+
+    [ants_decomposition]
+    x_split = 2
+    y_split = 2
+
+    [ants_logging]
+    enabled = True
+
+Key Points
+----------
+
+ * All five command line tools share the same ``<sources> --output <output>
+   [--ants-config <config>]`` interface via
+   :class:`ants.command_parse.AntsArgParser`.
+ * ``ancil_2anc`` is for fileformat translation, ``ancil_create_shapefile``
+   produces validity polygons, ``ancil_fill_n_merge`` combines and fills
+   data, ``ancil_general_regrid`` moves data onto a target horizontal grid
+   (and can carry out vertical regridding too), and ``ancil_vertical_regrid``
+   moves data onto a target set of vertical levels.
+ * Prefer ``--search-method kdtree`` on ``ancil_fill_n_merge`` and
+   ``ancil_general_regrid`` for consistency across UM and LFRic pipelines
+   (see :doc:`tutorial_merge_fill`).
+ * If you want to do a vertical regrid for UGrid target domains use the
+   UG-ANTS tooling instead.
+ * See the ``rose-stem/app/`` directory in the ANTS repository for complete,
+   runnable configurations of each of these tools.
+
+See Also
+--------
+
+ * :doc:`tutorial_load_save`
+ * :doc:`tutorial_regridding`
+ * :doc:`tutorial_merge_fill`
+ * :doc:`core_capabilities`

docs/source/tutorial_load_save.rst

@@ -0,0 +1,136 @@
+.. meta::
+   :description lang=en: Tutorial on loading and saving data with ANTS
+   :keywords: load, save, netcdf, ancil, tutorial
+   :property=og:locale: en_GB
+
+.. include:: common.txt
+
+.. Created with assitance of MetOffice Enterprise Copilot
+
+Loading and saving data with ANTS
+==================================
+
+Every ANTS based application starts by loading source data and a target grid,
+and ends by saving the resulting field(s) to disk. ANTS is built on top of
+|Iris|, and while you can use :py:func:`iris.load` and :py:func:`iris.save`
+directly, the :mod:`ants.io.load` and :mod:`ants.io.save` modules add
+capability that ancillary generation specifically needs. You should prefer the
+ANTS routines over the base Iris ones unless you have good reason not to.
+
+Why use the ANTS load and save routines?
+-----------------------------------------
+
+:mod:`ants.io.load` uses Iris to read the fileformats Iris supports (and
+some ANTS specific ones, see :mod:`ants.fileformats`), then additionally:
+
+1. derives the global/regional status of the input data and sets the
+   corresponding metadata on the cube,
+2. guesses bounds for latitude and longitude coordinates where a source has
+   none, and
+3. removes any ``forecast_reference_time`` and ``forecast_period``
+   coordinates, which are not usually wanted in ancillary fields.
+
+:mod:`ants.io.save` similarly extends :py:func:`iris.save` with support for
+the F03 UM ancillary fileformat and a UKCA specific flavour of NetCDF, neither
+of which Iris can write natively.
+
+Loading data
+------------
+
+Use :func:`ants.io.load.load` to load one or more cubes from a filepath (or
+list of filepaths):
+
+.. code-block:: python
+
+    import ants
+
+    source_cubes = ants.io.load.load("/path/to/source_file.nc")
+
+This returns a :class:`~iris.cube.CubeList`, even where the file contains a
+single field. If you know your source contains exactly one field you may
+prefer :func:`ants.io.load.load_cube`, which returns a single
+:class:`~iris.cube.Cube` and raises an exception if more than one match is
+found.
+
+Loading a target grid
+----------------------
+
+Most ANTS applications need a target grid to process or regrid data onto.
+:func:`ants.io.load.load_grid` loads a grid definition, without a data
+payload, from one or more files, merging horizontal and vertical components
+where necessary:
+
+.. code-block:: python
+
+    target_grid = ants.io.load.load_grid("/path/to/target_grid_namelist")
+
+This supports both CAP compliant Fortran namelist grid definition files (use
+this where you do not need a land sea mask as part of your processing) and
+any fileformat that Iris can read (use this where you need the land sea mask
+of the target field).
+
+Where a target's land sea mask is needed directly, for example to make output
+consistent with land or sea only data, use
+:func:`ants.io.load.load_landsea_mask` instead. This function accepts either
+a land binary mask field or a land fraction field:
+
+.. code-block:: python
+
+    # land_threshold is only used when the source is a land fraction field;
+    # fractions greater than this value are treated as land.
+    target_lsm = ants.io.load.load_landsea_mask(
+        "/path/to/target_lsm.nc", land_threshold=0.5
+    )
+
+Saving data
+-----------
+
+Saving to NetCDF is done via :func:`ants.io.save.netcdf`:
+
+.. code-block:: python
+
+    import ants.io.save as save
+
+    save.netcdf(source_cubes, "/path/to/output.nc")
+
+To save to an F03 UM ancillary file, use :func:`ants.io.save.ancil` instead.
+Every cube saved this way **must have a STASH code attribute** set, since this
+is used to identify the field in the ancillary file:
+
+.. code-block:: python
+
+    import iris.fileformats.pp
+
+    for cube in source_cubes:
+        cube.attributes["STASH"] = iris.fileformats.pp.STASH.from_msi("m01s00i030")
+
+    save.ancil(source_cubes, "/path/to/output_ancil")
+
+If your data is destined for UKCA, use :func:`ants.io.save.ukca_netcdf`
+instead, which applies UKCA specific conventions (compression, bounds,
+data type coercion and attribute renaming) on top of a standard NetCDF save:
+
+.. code-block:: python
+
+    save.ukca_netcdf(source_cubes, "/path/to/output_ukca.nc")
+
+Key Points
+----------
+
+ * Prefer :mod:`ants.io.load` and :mod:`ants.io.save` over calling Iris
+   directly - they add corrections and capability that ancillary generation
+   depends on.
+ * :func:`ants.io.save.ancil` requires every cube to have a ``STASH``
+   attribute; NetCDF and UKCA NetCDF saves do not.
+ * Use :func:`ants.io.load.load_grid` for a namelist based target grid, and
+   :func:`ants.io.load.load_landsea_mask` where the target's land sea mask is
+   needed directly.
+
+See Also
+--------
+
+ * :doc:`tutorial_regridding` for using a loaded target grid to regrid data.
+ * :doc:`ancil_2anc` for a command line tool that wraps a load/save round
+   trip to convert between fileformats.
+ * :doc:`appendixA_time_handling` for the time metadata expectations placed
+   on data being saved as an ancillary.