Upgrade macros and kgo (#445)

Co-authored-by: Jenny Hickson <61183013+jennyhickson@users.noreply.github.com>

Author: james-bruten-mo

source/Development/inputs.rst

@@ -150,9 +150,6 @@ Please see :ref:`this page <macros>` for further information.
 
 .. important::
 
-    If your development includes an upgrade macro, you **must** add the
-    ``macro`` keyword to your ticket.
-
     **Do not** apply the upgrade macro to your dev branch prior to the review
     process. Instead you must create a test branch. See :ref:`testing`.
 

source/Development/macros.rst

@@ -32,7 +32,8 @@ following locations:
         ``applications/<APPLICATION>/rose-meta/lfric-<APPLICATION>/versions.py``
 
         Variations on this theme occur, e.g. LFRic Apps science sections or
-        Components in LFRic Core
+        Components in LFRic Core. Metadata directories are also symlinked from a
+        top level ``rose-meta`` directory.
 
 
 Within the file a blank upgrade macro will typically look like this:
@@ -65,8 +66,8 @@ Example of an upgrade macro
 
 Developer Sally Smith wishes to add the logical ``l_bugfix`` to the ``&run_bl``
 namelist in the UM. To do this, she replaces the `XXXX` line in the upgrade
-macro with her ticket number and adds herself as the author. Within the Python
-function ``upgrade``, she adds the appropriate command to include the new
+macro with her PR or Issue number and adds herself as the author. Within the
+Python function ``upgrade``, she adds the appropriate command to include the new
 logical. The macro then looks like this:
 
 .. code-block:: python
@@ -151,31 +152,31 @@ and test branch, but potentially not a development branch.
 
 .. tip::
 
-    The wrapper script will read the ``dependencies.sh`` file in your LFRic
+    The wrapper script will read the ``dependencies.yaml`` file in your LFRic
     Apps working copy and will checkout a temporary copy of the LFRic Core
     source if required. Some Core metadata changes will also modify the Core
     rose apps. In this case make sure to also commit these changes back to the
     core branch.
 
 To add upgrade macros to LFRic the following steps can be followed:
 
-1. In your local LFRic Apps clone update the core source in ``dependencies.sh``
-   if you have LFRic Core changes.
+1. In your local LFRic Apps clone update the core source in
+   ``dependencies.yaml`` if you have LFRic Core changes.
 
 2. Add your upgrade macros. These **must** be added to the ``versions.py`` file
-   in the same directory as the metadata being changed.
+   in the same ``rose-meta`` directory as the metadata being changed.
 
 3. Run the Upgrade Macro script in a test branch(see :ref:`testing`). This is
    located in the `SimSys_Scripts github repo
-   <https://github.com/MetOffice/SimSys_Scripts>`__ (at meto an up to date
-   clone is available in $UMDIR/SimSys_Scripts). The syntax for running is:
+   <https://github.com/MetOffice/SimSys_Scripts>`__ (at the MetOffice an up to
+   date clone is available in $UMDIR/SimSys_Scripts). The syntax for running is:
 
-.. code-block:: shell
+   .. code-block:: shell
 
-    export CYLC_VERSION=8
+       export CYLC_VERSION=8
 
-    SimSys_Scripts/lfric_macros/apply_macros.py vnX.Y_tZZZZ \
-        [--apps=/path/to/apps] [--core=/path/to/core] [--jules=/path/to/jules]
+       SimSys_Scripts/lfric_macros/apply_macros.py vnX.Y_tZZZZ \
+           [--apps=/path/to/apps] [--core=/path/to/core] [--jules=/path/to/jules]
 
 .. important::
 
@@ -185,14 +186,9 @@ To add upgrade macros to LFRic the following steps can be followed:
 The Apps, Core and Jules options are paths to sources for each of these. Apps
 will default to the present location (so it is recommended to launch from an
 Apps working copy). Core and Jules will default to reading the
-``dependencies.sh`` file in the Apps source if not provided.
+``dependencies.yaml`` file in the Apps source if not provided.
 
 The ``vnXX.Y_tTTTT`` option must match the After Tag of your upgrade macro.
 When setting this, the version is the last released version of LFRic Apps. If
 it's a linked Apps-Core ticket, then set the ticket number as the one where
 the most metadata changes are being made.
-
-.. tip::
-
-    The apply_macros script requires python >= 3.9. At the Met Office this can
-    be achieved by ``module load scitools``.

source/Development/testing.rst

@@ -43,29 +43,42 @@ Bespoke:
 Test branches & Upgrade Macros
 ------------------------------
 
-There are a few cases where testing your change will require you to make
-changes to your branch that don't want committing to trunk. To do this you can
-create a test branch. This is a branch-of-branch from your development branch
-and allows you to make those changes in an isolated environment while leaving
-your original development clean.
+.. tip::
+
+    While we continue to use ``dev`` and ``test`` branch nomenclature from fcm,
+    in github these terms have no technical meaning and are simply a way to
+    distinguish between 2 branches.
+
+There are a few cases where testing your change will require you to make changes
+to your branch that don't want committing to trunk. To do this you can create a
+test branch. This is a branch-of-branch from your development branch and allows
+you to make those changes in an isolated environment while leaving your original
+development clean.
 
-To create one:
+To create a test branch:
 
 .. code-block:: shell
 
-    fcm bc -t test --bob testbranchname \
-    fcm:project.x_br/dev/yourname/devbranchname
+    git switch -c test_branch_name [<start_point>]
 
-Then check this out and use it for running any tests you'd like to carry out.
+If not provided ``start_point`` will default to your
+current branch.
 
 If using a test branch then do list this on your ticket and include the results
 of this testing alongside those from your dev branch.
 
 .. Note::
 
-    If your tests fail then you will need to make and commit the fixes to the
-    development branch and create a new test branch from that latest revision
-    to test them.
+    If you need further updates to the dev branch which require retesting on the
+    test branch, you can update the test branch by merging in the dev branch.
+
+    .. code-block::
+
+        git switch test_branch_name
+        git merge dev_branch_name
+
+    If testing upgrade macros however, you will likely need a new test branch,
+    as the macros can only be applied once.
 
 Macros
 ^^^^^^

source/Reviewers/howtocommit.rst

@@ -71,103 +71,92 @@ Then switch to the up to date branch, e.g.
 **If** the ticket includes metadata changes, upgrade macro changes or a new
 rose-stem app then you will need to upgrade the test-suite.
 
-.. dropdown:: versions.py
+Update the versions.py file
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-    ``versions.py`` contains a sequence of upgrade macros. Each macro contains
-    a ``BEFORE_TAG`` and an ``AFTER_TAG`` which should create a single chain,
-    starting at the last release and finishing with the ticket you are
-    committing. The tags have the format version_ticket, i.e.
-    ``vnXX.Y_tZZZZ``.
+``versions.py`` contains a sequence of upgrade macros. Each macro contains a
+``BEFORE_TAG`` and an ``AFTER_TAG`` which should create a single chain, starting
+at the last release and finishing with the ticket you are committing. The tags
+have the format version_ticket, i.e. ``vnXX.Y_tZZZZ``.
 
-    When resolving conflicts in this file make sure that the new macro being
-    added by your ticket is added to the end of the file. Modify the
-    ``BEFORE_TAG`` to match the ``AFTER_TAG`` of the previous macro in the
-    chain.
+When resolving conflicts in this file make sure that the new macro being added
+by your ticket is added to the end of the file. Modify the ``BEFORE_TAG`` to
+match the ``AFTER_TAG`` of the previous macro in the chain.
 
-    If this is the first macro since the release then the ``BEFORE_TAG`` will
-    be the version number with no added ticket number.
+If this is the first macro since the release then the ``BEFORE_TAG`` will be the
+version number with no added ticket number.
 
-    Remove the template macro if it is still present.
+Remove the template macro if it is still present.
 
-.. dropdown:: Applying Macros
+Apply the upgrade macros
+^^^^^^^^^^^^^^^^^^^^^^^^
 
-    To update the test suite for an upgrade macro, please run:
+To update the test suite for an upgrade macro, please run:
 
-    .. tab-set::
-
-        .. tab-item:: UM
-
-            .. code-block:: shell
-
-                ./admin/rose-stem/update_all.py \
-                    --path=/path/to/working/copy/of/trunk \
-                    --um=vnXX.Y_tZZZZ \
-                    [--jules-path=/path/to/working/copy/of/jules/trunk]
-
-            where ``-\-um=vnXX.Y_tZZZZ`` is the ``AFTER_TAG`` of the latest
-            upgrade macro.
-
-            If there is a macro for fcm_make or createbc then check that the
-            makes ``version*_*.py`` has the correct BEFORE and AFTER tags and
-            append ``-\-makeum=vnXX.Y_tZZZZ`` and/or
-            ``-\-createbc=vnXX.Y_tZZZZ`` to the above command.
+.. tab-set::
 
-            .. warning::
+    .. tab-item:: UM
 
-                Please ensure that Cylc7 is used with ``update_all.py`` @vn13.5.
+        .. code-block:: shell
 
-            .. note::
+            ./admin/rose-stem/update_all.py \
+                --path=/path/to/working/copy/of/trunk \
+                --um=vnXX.Y_tZZZZ \
+                [--jules-path=/path/to/working/copy/of/jules/trunk]
 
-                The ``-\-jules-path`` option is only required if there are
-                linked `jules-shared
-                <https://code.metoffice.gov.uk/trac/jules/browser/main/trunk/rose-meta/jules-shared>`__
-                metadata changes.
+        where ``-\-um=vnXX.Y_tZZZZ`` is the ``AFTER_TAG`` of the latest
+        upgrade macro.
 
-        .. tab-item:: JULES
+        If there is a macro for fcm_make or createbc then check that the
+        makes ``version*_*.py`` has the correct BEFORE and AFTER tags and
+        append ``-\-makeum=vnXX.Y_tZZZZ`` and/or
+        ``-\-createbc=vnXX.Y_tZZZZ`` to the above command.
 
-            .. code-block:: shell
+        .. note::
 
-                ./bin/upgrade_jules_test_apps vnX.Y_tZZZZ
+            The ``-\-jules-path`` option is only required if there are
+            linked `jules-shared
+            <https://code.metoffice.gov.uk/trac/jules/browser/main/trunk/rose-meta/jules-shared>`__
+            metadata changes.
 
-            where ``vnX.Y_tZZZZ`` is the ``AFTER_TAG`` of the latest upgrade
-            macro. The upgrade is expected to fail for the ``fab_jules``,
-            ``metadata_checker`` and ``umdp3_checker`` apps.
+    .. tab-item:: JULES
 
-        .. tab-item:: LFRic Apps + Core
+        .. code-block:: shell
 
-            .. code-block:: shell
+            ./bin/upgrade_jules_test_apps vnX.Y_tZZZZ
 
-                apply_macros.py vnX.Y_tZZZZ [--apps=/path/to/apps] \
-                    [--core=/path/to/core] [--jules=/path/to/jules]
+        where ``vnX.Y_tZZZZ`` is the ``AFTER_TAG`` of the latest upgrade
+        macro. The upgrade is expected to fail for the ``fab_jules``,
+        ``metadata_checker`` and ``umdp3_checker`` apps.
 
-            where ``vnX.Y_tZZZZ`` is the ``AFTER_TAG`` of the latest upgrade
-            macro and the others are paths to the relevant sources. Apps
-            defaults to the current location. Core and Jules default to
-            reading the ``dependencies.sh`` file in the Apps source. A copy of
-            ``apply_macros.py`` is available at
-            ``$UMDIR/SimSys_Scripts/lfric_macros``.
+    .. tab-item:: LFRic Apps + Core
 
-            .. tip::
+        .. code-block:: shell
 
-                ``module load scitools`` will give all required dependencies
-                for Met Office users.
+            apply_macros.py vnX.Y_tZZZZ [--apps=/path/to/apps] \
+                [--core=/path/to/core] [--jules=/path/to/jules]
 
-            .. note::
+        where ``vnX.Y_tZZZZ`` is the ``AFTER_TAG`` of the latest upgrade
+        macro and the others are paths to the relevant sources. Apps
+        defaults to the current location. Core and Jules default to
+        reading the ``dependencies.yaml`` file in the Apps source. A copy of
+        ``apply_macros.py`` is available at
+        ``$UMDIR/SimSys_Scripts/lfric_macros``.
 
-                All LFRic Core tickets with macros are expected to be linked
-                with LFRic Apps, though they may not have required an LFRic
-                Apps development branch (although an Apps ticket should be
-                provided). This is fine - if there is no LFRic Apps branch
-                just checkout the LFRic Apps main. Then run the apply_macros
-                script as described above and this will share the upgrade
-                macro across both LFRic Apps and LFRic Core as needed.
+        .. tip::
 
-    .. important::
+            ``module load scitools`` will give all required dependencies
+            for Met Office users.
 
-        Now commit the changes made by the macros script back to the developers
-        branch.
+        .. note::
 
-        Do not push the changes at this stage.
+            All LFRic Core tickets with macros are expected to be linked
+            with LFRic Apps, though they may not have required an LFRic
+            Apps development branch (although an Apps ticket should be
+            provided). This is fine - if there is no LFRic Apps branch
+            just checkout the LFRic Apps main. Then run the apply_macros
+            script as described above and this will share the upgrade
+            macro across both LFRic Apps and LFRic Core as needed.
 
 .. dropdown:: New rose-stem app?
 
@@ -214,6 +203,13 @@ rose-stem app then you will need to upgrade the test-suite.
     update the `table that lists them
     <https://code.metoffice.gov.uk/trac/um/wiki/TempUMlogicals>`__.
 
+.. important::
+
+    Now commit the changes made by the macros script back to the developers
+    branch.
+
+    Do not push the changes at this stage.
+
 3. Test (if no KGO)
 --------------------
 
@@ -361,8 +357,7 @@ KGO differently*
     is a one-time setup step required to allow you to generate KGO using the
     update script.
 
-    Edit ``~/.metomi/rose.conf`` on *all platforms* - Desktop, XCE/F, XCS and
-    EXZ to contain the following:
+    Edit ``~/.metomi/rose.conf`` on *all platforms* - VDI, EXAB, EXCD, EXZ
 
     .. code-block::
 
@@ -384,7 +379,7 @@ KGO differently*
         a script.
 
         #. Run the rose stem tasks that require a KGO update, plus any other
-           testing required (see above) - if unsure run the `all,ex1a`.
+           testing required (see above) - if unsure run the `all` group.
 
         .. code-block:: shell
 

source/WorkingPractices/gh_dev_init.rst

@@ -150,13 +150,15 @@ To create a branch and switch to it from the command line, the syntax is,
 
         .. code-block:: shell
 
-            # parent_branch will default to the current branch if not provided
+            # start_point is the point to create a branch from
+            # It can be eg. a branch, hash, tag
+            # It will default to the current branch if not provided
             # switch will automatically change to the newly created branch
-            git switch -c <branch_name> [<parent_branch>]
+            git switch -c <branch_name> [<start_point>]
 
             # Or
 
-            git branch <branch_name> <parent_branch>
+            git branch <branch_name> <start_point>
             git switch <branch_name>
 
 Developing a Change