diff --git a/source/Reviewers/curaterelease.rst b/source/Reviewers/curaterelease.rst index 1e3cb076..acbf2f54 100644 --- a/source/Reviewers/curaterelease.rst +++ b/source/Reviewers/curaterelease.rst @@ -22,65 +22,59 @@ Curating a Release .. _reference-tagging: -Tagging FCM Trunks ------------------- - -Tagging fcm trunks with new release keywords is done in lots of places -throughout these instructions. This section is a guide for doing this, using -the UM as an example. - -The keywords need to be added to the base directory of the project, i.e. in the -directory that contains trunk and branches. +Release Issue +------------- -.. code-block:: shell +Open a Release Curation Issue in the ``git_playground``, using the release +Issue template. - # Note the -q means quiet checkout, nothing is printed to std out. The -N - means - # only the top level directory is extracted, otherwise it would extract the - # entire repository (and take many hours!) - # The 'fcm log' command gives you the most recent log message for the - # respective trunk, and more importantly, the revision number. +.. _github-releases: - fcm co -q -N fcm:um.x um - fcm log -l1 fcm:um.x/trunk - cd um - fcm pe fcm:revision . +Stable + Main Release +--------------------- -In the editor that comes up (you may need to specify the editor using -``--editor-cmd vi``) add the new keyword, e.g. ``vn11.5 = 810`` (noting the -format may change away from the UM). Once completed save the changes and close -the editor. Finally commit the change, ``fcm ci``. +This section describes the process of merging a release branch onto main and +stable. It assumes that a branch has been created from main, and that any +changes required for the release have been committed to it and tested. This +section is relevant for all repositories with a stable and main branch setup. +.. note:: -Release Ticket --------------- + Some repos (Socrates, Casim) do not require release changes, so a PR should + just be opened to merge the ``main`` branch into the ``stable`` branch. Then + a second PR should be opened to merge ``stable`` back into ``main`` to + ensure ``main`` is never behind. Any tags required can then be made at this + point. -Open a UM X.Y release Curation Ticket, and assign tasks as a team, +The release process will be completed by 2 people with commit privilege to the +relevant repository, at least one of whom must be an ``admin``. One will have +developed the release branch and the other will review it (**developer** and +**reviewer** below). -.. code-block:: +* Once development and local testing has been completed, the **developer** + should open a PR, targetting the ``stable`` branch. +* The **reviewer** will then review and commit the branch. When committing the + branch, ensure that the merge method is ``merge``. This should be the default + for the ``stable`` branch as we want to keep the history of ``main`` in the + ``stable`` branch. +* The **developer** will then create a new PR, to merge the ``stable`` branch + into ``main``. +* At this point an admin will need to modify the branch protection rules for the + ``main`` branch, so that the commit can be performed with a normal merge. This + keeps ``main`` and ``stable`` with an identical history. - The following tickets are required to deliver UM vnX.Y: + * Navigate to the ``main`` ruleset. + * Disable ``Require linear history``. + * Set ``merge`` as an allowed merge strategy and disable ``squash``. - ||= Led by... =||= Description =||= Ticket # =|| - ||||||'''Pre release'''|| - || || Test release || || - || || Partner testing || || - || || Scientific Software Stack Update || || - ||||||'''Release'''|| - || || JULES umX.Y release - [jules:wiki:CuratingARelease] || || - || || Shumlib + Mule releases || || - || || Build and install the main release || || - || || LFRic Apps Release || || - ||||||'''Post Release'''|| - || || Release notes || || - || || Update standard suites || || - || || Check resource monitoring scripts still work || || - || || Install the Stash Browser || || - || || UMDP3 Release || || - || || Standard Jobs + Wiki Page || || - || || Review and update trunk and shared account permissions || || +* The **reviewer** can now ``merge`` the second PR. +* The admin **must** now revert the 2 settings above. +* Finally, the release can be created and tagged, - [https://metoffice.github.io/simulation-systems/Reviewers/curaterelease.html Curating a release Page] + * From the github repo, select ``releases`` and then ``Draft a new release``. + * Create a new tag and title the release with the same name, eg. ``vn14.0``. + * Select to ``Generate release notes``. + * Then ``Publish release``. Pre-Release @@ -198,10 +192,3 @@ UM + Apps Releases **Dependencies** None - - -Mid-Release Tasks ------------------ - -:ref:`Mid Release Prebuilds` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/source/Reviewers/releases/um_main_release.rst b/source/Reviewers/releases/um_main_release.rst index f8ca2561..d087281f 100644 --- a/source/Reviewers/releases/um_main_release.rst +++ b/source/Reviewers/releases/um_main_release.rst @@ -3,46 +3,33 @@ UM Main Release =============== -Create a ticket and Create Branches ------------------------------------ - -To document the process create a ticket and put a link to it on the release -curation ticket. You may wish to include links to the UM and UM metadata -branches required and a wiki page for the reviews. - -.. code-block:: - - Documents the 'Build and install the
UM release' - - NOTES: Any changes required that are not a direct instruction from this section of the guide. - - '''Branch :''' [log:main/branches/dev/branch_path_n_name dev/branch_path_n_name] - '''Meta Branch :''' [log:meta/branches/dev/branch_path_n_name dev/branch_path_n_name] - '''[wiki:ticket/ticket_no/CodeSystemReview Code/System Review]''' +Create Branches +---------------------- +Ensure you have a :ref:`fork ` of both the ``um`` and ``um_meta`` +repositories, and that the ``main`` branches in each are up to date with the +upstream repository. -Create and check out both a head of trunk UM branch and a head of trunk UM meta -branch. Then update the ticket description. +In a clone of these forks, :ref:`create a branch ` using the +``main`` branch as the parent. In the UM branch, update the ``um_meta`` entry in +the ``dependencies.yaml`` file to point at your metadata branch. -.. code-block:: shell +.. important:: - fcm bc -k ticket_no vn11.5_um_release fcm:um.x_tr - fcm co fcm:um.x_br/dev/username/r12345_vn11.5_um_release - fcm bc -k ticket_no vn11.5_meta_release fcm:um_meta.x_tr - fcm co fcm:um_meta.x_br/dev/username/r12345_vn11.5_meta_release + Ensure you create branches from main, otherwise you will not include the + changes from the past release. Tagging Feeder Trunks --------------------- -* :ref:`Tag ` the head of the feeder repositories with - keywords for the new UM version if they do not already exist. +* Add a ``umX.Y`` tag to each of the feeder repositories -* ``fcm:casim.x`` -* ``fcm:jules.x`` -* ``fcm:shumlib.x`` -* ``fcm:socrates.x`` -* ``fcm:ukca.x`` + * Casim + * Jules + * Shumlib + * Socrates + * UKCA Send an email to all repository owners to let them know that the the head of the trunk has been tagged. @@ -52,6 +39,10 @@ the trunk has been tagged. Jules should already have been done by the Jules release, but this is worth checking. +For each of these repositories, modify the ``ref`` in the UM +``dependencies.yaml`` file, to point at the new ``umX.Y`` tag. Commit this +change to the branch. + Apply Code Styling ------------------ @@ -59,25 +50,33 @@ Apply Code Styling * Switch to the UM branch * Set ``export RUNFULL=1`` then run ./admin/code_styling/apply_styling * If any files have changed, check nothing has gone wrong with a quick compile - ``rose stem --group=fcm_make_ex1a_gnu_um_rigorous_omp`` + ``cylc vip -z g=fcm_make_ex1a_gnu_um_rigorous_omp -n um_release_style_check + ./rose-stem`` * Commit any changes Checking Metadata and Rose Apps ------------------------------- -The JULES release must be completed first and all jules-shared metadata changes -from the JULES repository must be centrally installed before progressing. +.. important:: + + The JULES release must be completed first and all jules-shared metadata + changes from the JULES repository must be centrally installed before + progressing. * First switch to the UM branch. * Check that the metadata meets the Rose standards: run ``rose config-dump -C - rose-meta``. Do this before running release_new_version, so that any + rose-meta``. Do this before running ``release_new_version``, so that any metadata errors are fixed before the new vnX.Y metadata directories are created, otherwise you'll have to check both vnX.Y and HEAD. - * Run ``fcm diff`` on HEAD. Are the changes sensible? They often just - involve moving sections of meta-data to be in the correct alphabetical - order. However, `UM:#1824 + * Run ``git difftool origin/main``. Are the changes sensible? They often + just involve moving sections of meta-data to be in the correct + alphabetical order. + + .. important:: + + `UM:#1824 `__ added comments for some additional triggers ([43853]) to circumvent a bug in Rose. Running config-dump will move the location of these comments to the @@ -90,7 +89,7 @@ from the JULES repository must be centrally installed before progressing. atmos and fcm-make apps, so they should be correct. Are there any other changes (e.g. to rose-ana apps)? Are they understood? Commit them to the branch or discuss with the team as appropriate. -* Commit all changes before moving onto the next section +* Commit all changes before moving onto the next section. Running the Release Script @@ -101,47 +100,40 @@ version script will be run from. The syntax is, .. code-block:: shell - ../admin/rose-stem/release_new_version.py -c -n -j - eg. ../admin/rose-stem/release_new_version.py -c 13.4 -n 13.5 -j 7.5 + ../admin/rose-stem/release_new_version.py -c -n -j + eg. ../admin/rose-stem/release_new_version.py -c 13.4 -n 13.5 -j 7.5 * Open a new terminal and inspect that the version number update macro added by the script is correct, and that the tXXXX template macro has been deleted appropriately. - * This has caused problems before see `this edge case - `_. - The upgrade macro should fail to execute if the macro chain is - incorrect, as it won't be able to upgrade an app to the new version - - this is likely this edge case. + * This has caused problems before see `this edge case + `_. + The upgrade macro should fail to execute if the macro chain is + incorrect, as it won't be able to upgrade an app to the new version - + this is likely this edge case. Copying the metadata and upgrade macros to the um_meta branch ------------------------------------------------------------- The next step is to move the macros and metadata into the meta branch. The -metadata will have been created already by the release script. Below shows an -example of the commands run to move from 11.4 to 11.5, from the top directory -of the working copy of the UM branch, +metadata will have been created already by the release script. For each of the +``um-atmos``, ``um-fcm-make`` and ``um-createbc`` metadata sections, move the +new ``vnX.Y`` directory into the ``um_meta`` clone, eg. .. code-block:: shell - version="version114_115.py" - vn="vn11.5" - path="/path/to/meta/working_copy" - - fcm mv rose-meta/um-atmos/$version $path/um-atmos/$version - fcm mv rose-meta/um-fcm-make/$version $path/um-fcm-make/$version - fcm mv rose-meta/um-createbc/$version $path/um-createbc/$version - - fcm mv rose-meta/um-atmos/$vn $path/um-atmos/$vn - fcm mv rose-meta/um-fcm-make/$vn $path/um-fcm-make/$vn - fcm mv rose-meta/um-createbc/$vn $path/um-createbc/$vn + mv rose-meta/um-atmos/vnX.Y /path/to/meta_clone/um-atmos/ + cd /path/to/meta_clone + git add um-atmos/vnX.Y -Note: there is no need to move um-crmstyle as it only contains HEAD metadata. +Note, there is no need to move um-crmstyle as it only contains HEAD metadata. -Manually add a line to each of the ``um-atmos/versions.py``, -``um-fcm-make/versions.py`` and ``um-createbc/versions.py`` files in the meta -branch to import the newly copied versionXX_XY.py file. +In the ``um_meta`` clone, manually add a line to each of the +``um-atmos/versions.py``, ``um-fcm-make/versions.py`` and +``um-createbc/versions.py`` files to import the newly copied ``versionXX_XY.py`` +file (see the existing line). Commit the changes to both the UM and Meta branches. @@ -151,41 +143,22 @@ Final Checks **UM AUX Changes** -If there are changes to the AUX trunk in this release, are we picking up the -head of the AUX trunk (fcm:um_aux)? A new keyword will need to be created and -copied into the rose-stem/rose-suite.conf file. - -.. code-block:: shell - - fcm co -q -N fcm:um_aux.x aux - fcm log -l1 fcm:um_aux.x/trunk - cd aux - fcm pe fcm:revision . - fcm commit - -.. warning:: - - Updating ``HOST_SOURCE_UM_AUX`` with the new keyword is NOT done - automatically by release_new_version.py as it doesn't need to be done - every release +If there are changes to the AUX trunk in this release, then add a new tag to the +``um_aux`` repository and then update the ``ref`` in the UM +``dependencies.yaml`` file - commit this to the branch. **Other Points** -* Make sure the prebuilds are set to ``true`` in the - ``site/meto/variables.cylc`` by checking the line, ``{% do SITE_VARS.update( - {"PREBUILDS" : true}) %}`` * Check rose-stem/rose-suite.conf? - * Are the UM, JULES, SOCRATES, CASIM and UKCA versions correct? These - should be the keywords setup earlier. - * Is housekeeping ``true``? - * Are the KGO versions correct in the ``variables.cylc`` file for each - site? - * Does the minimum version of Rose/Cylc need to be increased? (Do any - rose-ana changes require new functionality?) - * Do any of the apps or parts of the suites reference ``$UMDIR`` - they - shouldn't (the correct thing to do is to reference - ``$UM_INSTALL_DIR``). + * Is housekeeping ``true``? + * Are the KGO versions correct in the ``variables.cylc`` file for each + site? + * Does the minimum version of Rose/Cylc need to be increased? (Do any + rose-ana changes require new functionality?) + * Do any of the apps or parts of the suites reference ``$UMDIR`` - they + shouldn't (the correct thing to do is to reference + ``$UM_INSTALL_DIR``). * ``grep`` for any instances of the old version keyword(s). Fix as required and add any corrections to the instructions on this page too. @@ -205,54 +178,38 @@ Preparing to Test exists in UMDIR on **all** platforms - this should have been done as part of the test release. - * If not, rename the inputs directory ``$UMDIR/standard_jobs/inputs/vnX.Y`` - to the new version number and be sure to symlink the previous version to - it. Do this all on one line to minimise any glitches during the rename. - This needs to be repeated on all platforms. i.e. to update from vn11.5 - to vn11.6 one would run, ``mv vn11.5 vn11.6; ln -s vn11.6 vn11.5``. - -* Local keywords for the UM should be put in your ``~/.metomi/fcm/keyword.cfg`` - file on **all** platforms (don't forget to remove them afterwards). The tag - should correspond to the version you are releasing and the version number - should be the revision of the trunk from which you branched. For example: - - .. code-block:: - - revision[um.x:vn10.0] = 112 - revision[um.xm:vn10.0] = 112 + * If not, rename the inputs directory ``$UMDIR/standard_jobs/inputs/vnX.Y`` + to the new version number and be sure to symlink the previous version to + it. Do this all on one line to minimise any glitches during the rename. + This needs to be repeated on all platforms. i.e. to update from vn11.5 + to vn11.6 one would run, ``mv vn11.5 vn11.6; ln -s vn11.6 vn11.5``. * For the rose_ana tasks to pass new KGO also needs to be generated for the new version, since you are about to run the ``all`` group test anyway you should use this opportunity to produce a new set of KGO. - * KGO is installed using the scripts in SimSys_Scripts. In order for the - script to work you must first change the KGO directories in the - ``variables.cylc`` and platform-specific ``variables_PLATFORM.cylc`` - files back to whichever versions were present before the - ``release_new_version.py`` script was run - you can do this with a - simple copy from the head of the trunk. Be careful to ensure this is - only changing the KGO versions for each variable as expected. **DO NOT - COMMIT this change - you will be reverting it later**. - - .. code-block:: shell + * KGO is installed using the scripts in SimSys_Scripts. In order for the + script to work you must first change the KGO directories in the + ``variables.cylc`` and platform-specific ``variables_PLATFORM.cylc`` + files back to whichever versions were present before the + ``release_new_version.py`` script was run - you can do this with a + simple copy from the head of main. Be careful to ensure this is + only changing the KGO versions for each variable as expected. **DO NOT + COMMIT this change - you will be reverting it later**. - fcm export --force fcm:um.x_tr/rose-stem/site/meto/variables.cylc \ - rose-stem/site/meto/ + .. code-block:: shell - fcm export --force fcm:um.x_tr/rose-stem/site/meto/ \ - variables_azspice.cylc rose-stem/site/meto/ + git clone git@github.com:MetOffice/um.git um_main - fcm export --force \ - fcm:um.x_tr/rose-stem/site/meto/variables_ex1a.cylc \z - rose-stem/site/meto/ + cp um_main/rose-stem/site/meto/variables* /path/to/um/branch/rose-stem/site/meto/ - * Current KGO files will have the older UM version in the fixed length - header and lookups. In order for the rose-ana tasks that use mule-cumf - to not give false rose-ana failures we must temporarily ignore the model - version. There is some logic in the UM rose stem suite to enable this. - Open your ``~/.metomi/rose.conf`` file, on **all** platforms, and add - the following lines to the rose-ana section, making sure that - bypass-version-check is true: + * Current KGO files will have the older UM version in the fixed length + header and lookups. In order for the rose-ana tasks that use mule-cumf + to not give false rose-ana failures we must temporarily ignore the model + version. There is some logic in the UM rose stem suite to enable this. + Open your ``~/.metomi/rose.conf`` file, on **all** platforms, and add + the following lines to the rose-ana section, making sure that + bypass-version-check is true: .. code-block:: @@ -268,10 +225,7 @@ not to forget the source argument to the UM metadata branch, .. code-block:: shell - rose stem --task=all -S PREBUILDS=false -S HOUSEKEEPING=false \ - -S USE_EXAB=true --source=. --source=/path/to/metadata/working_copy - - cylc play + cylc vip -z g=all -S HOUSEKEEPING=false -S USE_EXAB=true -n um_release_vnx.y ./rose-stem Before continuing the next step you should make sure the suite has run as expected. All tests should pass apart from any tasks that output netcdf @@ -281,14 +235,14 @@ version from the comparison as we did with tests that use mule-cumf. .. tip:: - Check the test results by running something like + Check the test results by running something like - .. code-block:: shell + .. code-block:: shell - find ~cylc-run//runN/log/job -path "*rose_ana*" -type f \ - -name job.status \ - | xargs grep -l CYLC_JOB_EXIT=ERR \ - | grep -vE "(scm|netcdf)" + find ~cylc-run//runN/log/job -path "*rose_ana*" -type f \ + -name job.status \ + | xargs grep -l CYLC_JOB_EXIT=ERR \ + | grep -vE "(scm|netcdf)" The ``meto_update_kgo.sh`` script is stored in SimSys_Scripts. As yourself, navigate to ``$UMDIR/SimSys_Scripts/kgo_updates`` directory and run @@ -306,77 +260,69 @@ navigate to ``$UMDIR/SimSys_Scripts/kgo_updates`` directory and run with the name ``vnX.Y`` where this is the new version number. * It will show you the settings to double check before you continue. - * Pay particular attention to the preview of the list of commands the - script will present you with to ensure it has accounted for all expected - KGO files. + * Pay particular attention to the preview of the list of commands the + script will present you with to ensure it has accounted for all expected + KGO files. * The script will install the new kgo on every platform in order azspice->ex1a. Once these are finished installing it will rsync to the EXCD and EXZ. To install the entire kgo database will take some time. +.. important:: + + The authentication for running as the shared user via ``sudo`` will likely + have expired before each of the 2 rsync commands, so pay attention as the + script nears these points. If you forget and the command times out, the + command can be launched manually from the command line. Ask the team if + for help if required. + Once you believe you have installed the KGO you should fcm revert the changes -you made to the variables*.cylc files to reset the KGO variables, ``fcm revert +you made to the variables*.cylc files to reset the KGO variables, ``git restore rose-stem/site/meto/variables*`` +.. tip:: + + Has the ability to reload the test suite been enabled yet? If so ``cylc + vr`` can likely be used to restart the original suite. + + This is likely the case after moving to github - try and update these WPs if + so. + The test suite should now be rerun to confirm the kgo has been installed properly. As we can't restart Cylc8 rose-stem suites, the entire thing needs to be rerun. We're just checking that the kgo has been installed, so it's probably unnecessary to wait for the entire thing - instead just ensure a reasonable range of rose-ana tasks have passed. -.. tip:: - - Has the ability to reload the test suite been enabled yet? If so ``cylc - vr`` can likely be used to restart the original suite. These instructions - also need updating! - Review and Commit ----------------- -Ensure all changes are committed to both branches and then pass along for a -review to someone in the team. +Ensure all changes are committed to both branches and then create a PR for each +of the ``um`` and ``um_meta`` branches and pass along for a review and commit. -Notes for Reviewer: - -* In ``rose-stem/site/meto/variables``, ensure the ``PREBUILDS`` variable near - the top is set to true. -* Once happy, commit both the meta and main branches, and return the ticket to - the developer. - -Now :ref:`tag ` the trunk with the ``vnX.Y = RRR`` tag. - -**Now make sure to revert changes to ``~/.metomi/fcm/keyword.cfg`` on all -platforms** +You and the reviewer should work through the process of committing the branches +together - :ref:`see here for details page`. Install the Release ------------------- +First get a local clone of the head of the ``um`` repository, now that the +release has been committed. This makes it available to the shared account +without having to rely on the mirrors. + The main installation of ctldata, utilities and prebuilds can now take place. This all takes place as the ``umadmin`` account so log in to that now. -Delete any remaining temporary vnX.Y keywords for umadmin/umtest, on **all** -platforms. Check all keyword.cfg files, and do both accounts now. They could -be left over from the earlier test build, even if you didn't set them. - -Check out the UM trunk into a working copy. ``umadmin`` can only check out from -the mirror. - -.. code-block:: shell - - fcm co fcm:um.xm_tr@vnX.Y umX.Y_install - cd umX.Y_install +Copy the ``um`` clone you created just now. First check that the upgrade has gone successfully and the new install will appear in the correct place. Do this by running, .. code-block:: shell - rose stem --group=install -S CENTRAL_INSTALL=false -S PREBUILDS=false \ - -S USE_EXAB=true - - cylc play + cylc vip -z g=install -S CENTRAL_INSTALL=false -S USE_EXAB=true -z umx.y_install ./rose-stem and check that ``~umadmin/cylc_run//runN/share/vnX.Y`` exists and is the new version number. If that has worked, change the @@ -384,77 +330,27 @@ CENTRALL_INSTALL flag to true and rerun, .. code-block:: shell - rose stem --group=install -S CENTRAL_INSTALL=true -S PREBUILDS=false \ - -S USE_EXAB=true - - cylc play + cylc vip -z g=install -S CENTRAL_INSTALL=true -S USE_EXAB=true -z umx.y_install ./rose-stem Next, rerun the install for the 2nd host zone, .. code-block:: shell - rose stem --group=ex1a_install -S CENTRAL_INSTALL=true \ - -S PREBUILDS=false -S USE_EXCD=true - - cylc play + cylc vip -z g=ex1a_install -S CENTRAL_INSTALL=true -S USE_EXCD=true -z umx.y_install ./rose-stem Finally, rerun the install for the EXZ, .. code-block:: shell - rose stem --group=ex1a_install -S CENTRAL_INSTALL=true \ - -S PREBUILDS=false -S USE_EXZ=true - - cylc play + cylc vip -z g=ex1a_install -S CENTRAL_INSTALL=true -S USE_EXZ=true -z umx.y_install ./rose-stem The release is now installed and can be announced. -Make Release Prebuilds ----------------------- - -Now it is time to install the prebuilds. - -.. important:: - - Use Cylc 7 (``export CYLC_VERSION=7``) to install the prebuilds. It is - important to set the source to the UM fcm mirror in the commands below, - and use the config option to point at the rose-stem directory. If this - wasn't done, prebuild availability would depend on the host machine you - are currently on being available. rose-stem in cylc8 doesn't support this, - hence using cylc7. - - A fix for this will likely become available with the move to git. The - timescales for that are shorter than for removing Cylc7. - -First install the prebuilds on Azure Spice and EXAB, - -.. code-block:: shell - - export CYLC_VERSION=7 - rose stem --group=prebuilds --source=fcm:um.xm_tr@vnX.Y \ - --name=vnX.Y_prebuilds --config=./rose-stem \ - -S MAKE_PREBUILDS=true -S USE_EXAB=true - -And then on the EXCD - make sure to **not** use ``--new`` in this command or -the previous set will have been overwritten. - -.. code-block:: shell - - export CYLC_VERSION=7 - rose stem --group=ex1a_fcm_make,ex1a_fcm_make_portio2b \ - --source=fcm:um.xm_tr@vnX.Y --name=vnX.Y_prebuilds \ - --config=./rose-stem -S MAKE_PREBUILDS=true -S USE_EXCD=true - -And finally on the EXZ - make sure to **not** use ``--new`` in this command or -the previous set will have been overwritten. - -.. code-block:: shell +.. tip:: - export CYLC_VERSION=7 - rose stem --group=ex1a_fcm_make,ex1a_fcm_make_portio2b \ - --source=fcm:um.xm_tr@vnX.Y --name=vnX.Y_prebuilds \ - --config=./rose-stem -S MAKE_PREBUILDS=true -S USE_EXZ=true + This is the point at which prebuilds used to be created. If prebuilds become + possible again, update the working practices here. Monsoon Installation @@ -467,12 +363,12 @@ Monsoon Installation We also install the UM onto Monsoon - to do this you will need a Monsoon account with access to the ``umadmin.mon`` shared account. -First, log into Monsoon as ``umadmin.mon`` and then check out the trunk at the -new version just released. +First, log into Monsoon as ``umadmin.mon`` and then clone the ``um`` repo. -.. code-block:: shell +.. note:: - fcm co fcm:um.xm_tr@vnXX.Y + At time of writing, the solution for extracting from github on Monsoon has + not been announced. Next, symlink the input data as was done for other platforms, @@ -484,30 +380,22 @@ Now run the central install group, .. code-block:: shell - rose stem --group=ex1a_install -S CENTRAL_INSTALL=true -S PREBUILDS=false - cylc play + cylc vip -z g=ex1a_install -S CENTRAL_INSTALL=true -z umx.y_install ./rose-stem Install prebuilds on Monsoon. Note the ``--no-run-name`` is required to force the install location to be consistent with other platforms. -.. code-block:: shell - - rose stem --group=ex1a_fcm_make,ex1a_fcm_make_portio2b \ - -S MAKE_PREBUILDS=true -n vnX.Y_prebuilds --no-run-name - cylc play - Finally we need to install the kgo for the release. Do this by running the ``ex1a`` group. Once that is finished, run the kgo install script(sourced from the SimSys_Scripts repo). .. code-block:: shell - rose stem --group=ex1a - cylc play + cylc vip -z g=ex1a -n umx.y_kgo ./rose-stem + # Wait for tests to complete python3 SimSys_Scripts/kgo_updates/kgo_update/kgo_update.py -N vnX.Y \ -P ex1a --new-release --non-interactive Check that the kgo has been installed in place correctly at ``$UMDIR/standard_jobs/kgo``. - diff --git a/source/Reviewers/releases/um_test_release.rst b/source/Reviewers/releases/um_test_release.rst index f3c67ca0..fd5a833a 100644 --- a/source/Reviewers/releases/um_test_release.rst +++ b/source/Reviewers/releases/um_test_release.rst @@ -11,7 +11,7 @@ UM Test Release Some of the test release only needs to be run on 1 HPC Host Zone - the instructions for this assume EXAB, but EXCD can be used instead. Other parts of the test release set up files for the main release - this needs - to be done on both EXAB and EXCD. + to be done on both EXAB EXCD and EXZ. Preparing Test Release Branches and Keyword Files ------------------------------------------------- @@ -19,51 +19,20 @@ Preparing Test Release Branches and Keyword Files This will involve switching between your own and the umadmin account on Azure Spice and EXAB. -* As yourself, create a UM ticket for the test release, marked 'Not for - Builds'. -* As yourself, create and check out a UM branch at the head of the trunk - (or whichever revision of the trunk you wish to do a test release with). -* As yourself, create and check out a head of trunk meta branch - (``fcm:um_meta.x_tr@head``). -* As umadin, setup the ``~/.metomi/fcm/keyword.cfg`` file on Azure Spice to - specify custom keywords for the UM, JULES, SOCRATES, CASIM, Shumlib, UKCA - and the mirrors of each of these named after the upcoming version and - pointing to the current head of trunk revisions of each project. No keywords - are needed for mule, moci or meta repositories. -* The general format looks like this- substitute model version and revisions - - * Remember to remove any existing hashtags/make sure these lines aren't - commented out - * For the test release we also need to set the Jules vn keyword(last line). - This needs to match the jules version that is passed to - release_new_version script below. - -.. code-block:: - - revision[um.x:vn13.5] = 123059 - revision[jules.x:um13.5] = 28020 - revision[socrates.x:um13.5] = 1563 - revision[casim.x:um13.5] = 10951 - revision[shumlib.x:um13.5] = 7324 - revision[ukca.x:um13.5] = 3196 - - revision[um.xm:vn13.5] = 123059 - revision[jules.xm:um13.5] = 28020 - revision[socrates.xm:um13.5] = 1563 - revision[casim.xm:um13.5] = 10951 - revision[shumlib.xm:um13.5] = 7324 - revision[ukca.xm:um13.5] = 3196 - - revision[jules.xm:vn7.5] = 28020 - -* As umadmin, scp this file to the EXAB, ``scp ~/.metomi/fcm/keyword.cfg - login.exab.sc:.metomi/fcm/keyword.cfg`` -* As yourself, copy this file into your own homespace ready for testing later, - ``cp ~umadmin/.metomi/fcm/keyword.cfg ~/.metomi/fcm/keyword.cfg``. Do this - on Azure Spice and EXAB. +* As yourself, create a UM Issue to track the test release. +* Ensure you have a :ref:`fork ` of both the ``um`` and ``um_meta`` + repositories, and that the ``main`` branches in each are up to date with the + upstream repository. +* In a clone of these forks, :ref:`create a branch ` using the + ``main`` branch as the parent. In the UM branch, update the ``um_meta`` entry + in the ``dependencies.yaml`` file to point at your metadata branch. Add a link + to both branches from your issue. +* In the UM branch ``dependencies.yaml`` file, update the ``um_meta`` entry to + point at your new branch. Update all other entries to point at ``main``. This + will use the Heads of each of those repositories for testing. * As yourself, add the following to ``~/.metomi/rose.conf`` on Azure Spice and EXAB. This prevents kgo failures due to the new version number when we test - later. Again, do this on Azure Spice and EXAB. + later. .. code-block:: @@ -79,18 +48,23 @@ release new version script will be run from. The syntax is, .. code-block:: shell - ../admin/rose-stem/release_new_version.py -c -n -j - eg. ../admin/rose-stem/release_new_version.py -c 13.4 -n 13.5 -j 7.5 + ../admin/rose-stem/release_new_version.py -c -n -j --test + eg. ../admin/rose-stem/release_new_version.py -c 13.4 -n 13.5 -j 7.5 --test + +.. note:: + + Adding ``--test`` will prevent the script from updating the Jules version + in the metadata which is needed as the new Jules tag will not yet exist. * Open a new terminal and inspect that the version number update macro added by the script is correct, and that the tXXXX template macro has been deleted appropriately. - * This has caused problems before see `this edge case - `. - The upgrade macro should fail to execute if the macro chain is - incorrect, as it won't be able to upgrade an app to the new version - - this is likely this edge case. + * This has caused problems before see `this edge case + `. + The upgrade macro should fail to execute if the macro chain is + incorrect, as it won't be able to upgrade an app to the new version - + this is likely this edge case. * As there is no kgo installed for the new version revert the changes to ``rose-stem/site/meto/variables_*.cylc`` as well as the ``BASE`` variable at @@ -99,30 +73,23 @@ release new version script will be run from. The syntax is, Copying the metadata and upgrade macros to the um_meta branch ------------------------------------------------------------- -As yourself, the next step is to move the macros and metadata into the meta -branch. The metadata will have been created already by the release script. -Below shows an example of the commands run to move from 11.4 to 11.5, from the -top directory of the working copy of the UM branch, +The next step is to move the macros and metadata into the meta branch. The +metadata will have been created already by the release script. For each of the +``um-atmos``, ``um-fcm-make`` and ``um-createbc`` metadata sections, move the +new ``vnX.Y`` directory into the ``um_meta`` clone, eg. .. code-block:: shell - version="version114_115.py" - vn="vn11.5" - path="/path/to/meta/working_copy" + mv rose-meta/um-atmos/vnX.Y /path/to/meta_clone/um-atmos/ + cd /path/to/meta_clone + git add um-atmos/vnX.Y - fcm mv rose-meta/um-atmos/$version $path/um-atmos/$version - fcm mv rose-meta/um-fcm-make/$version $path/um-fcm-make/$version - fcm mv rose-meta/um-createbc/$version $path/um-createbc/$version +Note, there is no need to move um-crmstyle as it only contains HEAD metadata. - fcm mv rose-meta/um-atmos/$vn $path/um-atmos/$vn - fcm mv rose-meta/um-fcm-make/$vn $path/um-fcm-make/$vn - fcm mv rose-meta/um-createbc/$vn $path/um-createbc/$vn - -Note: there is no need to move um-crmstyle as it only contains HEAD metadata. - -Manually add a line to each of the ``um-atmos/versions.py``, -``um-fcm-make/versions.py`` and ``um-createbc/versions.py`` files in the meta -branch to import the newly copied ``versionXX_XY.py`` file. +In the ``um_meta`` clone, manually add a line to each of the +``um-atmos/versions.py``, ``um-fcm-make/versions.py`` and +``um-createbc/versions.py`` files to import the newly copied ``versionXX_XY.py`` +file (see the existing line). Commit the changes to both the UM and Meta branches. @@ -132,46 +99,23 @@ Installing Ctldata, Utilities and Prebuilds These steps are all done as umadmin -Check out the UM trunk into a working copy. umadmin can only check out from the -mirror - if immediately following the previous steps, ensure the mirror has -updated. - -.. code-block:: shell - - fcm co fcm:um.xm_tr@vnX.Y umX.Y_install - cd umX.Y_install +Take a copy of the ``um`` and ``um_meta`` branches. umadmin can only access the +mirrors so this may be more easily done by copying your own clones. First check that the upgrade has gone successfully and the new install will appear in the correct place. Do this by running, .. code-block:: shell - rose stem --group=install rose-stem -S CENTRAL_INSTALL=false \ - -S PREBUILDS=false -S USE_EXAB=true - cylc play + cylc vip -z g=install -S CENTRAL_INSTALL=false -S USE_EXAB=true -n umX.Y_test_release ./rose-stem -and check that ``~umadmin/cylc_run//runN/share/vnX.Y`` -exists and is the new version number. If that has worked, change the -CENTRALL_INSTALL flag to true and rerun, +and check that ``cylc_run//runN/share/vnX.Y`` exists and is +the new version number. If that has worked, change the ``CENTRALL_INSTALL`` flag +to true and rerun, .. code-block:: shell - rose stem --group=install rose-stem -S CENTRAL_INSTALL=true \ - -S PREBUILDS=false -S USE_EXAB=true - cylc play - -Now install the prebuilds by running, - -.. code-block:: shell - - rose stem --group=prebuilds -S MAKE_PREBUILDS=true \ - --workflow-name=vnX.Y_prebuilds --no-run-name - -.. tip:: - - In the main release, we use cylc7 for the prebuild install as the Cylc8 - rose-stem is missing a feature. As we are going to be removing these - prebuilds shortly, the default Cylc8 is fine for the test release. + cylc vip -z g=install -S CENTRAL_INSTALL=true -S USE_EXAB=true -n umX.Y_test_release ./rose-stem Navigate to the input data directory on azure spice (``$UMDIR/standard_jobs/inputs``) and run the following command which copies @@ -188,14 +132,12 @@ Repeat this step on **all of** EXAB, EXCD and EXZ. Test the Branch --------------- -These steps are done as yourself. In your UM branch working copy, ensure the -``PREBUILDS`` variable in ``rose-stem/site/meto/variables.cylc`` is set to -true so we test the new prebuilds. Then run the entire test suite, +These steps are done as yourself. In your UM branch clone ensure the meta branch +is set in the ``dependencies.yaml`` file and then run the entire test suite, .. code-block:: shell - rose stem --group=all --source=. --source=/path/to/meta/working/copy - cylc play + cylc vip -z g=all -S USE_EXAB=true -n umX.Y_test_release ./rose-stem Before continuing the next step you should make sure the suite has run as expected. All tests should pass apart from any tasks that output netcdf @@ -205,23 +147,27 @@ version from the comparison as we did with tests that use mule-cumf. .. tip:: - Check the test results by running something like + Check the test results by running something like - .. code-block:: + .. code-block:: - find ~cylc-run//runN/log/job -path "*rose_ana*" -type f \ - -name job.status \ - | xargs grep -l CYLC_JOB_EXIT=ERR \ - | grep -vE "(scm|netcdf) + find ~cylc-run//runN/log/job -path "*rose_ana*" -type f \ + -name job.status \ + | xargs grep -l CYLC_JOB_EXIT=ERR \ + | grep -vE "(scm|netcdf) Test on Monsoon --------------- It's also sensible to check now that nothing has broken on Monsoon. Do this by -copying your keyword settings across to your account on there. Then check out -the UM main and meta branches and run the test suite as you in the previous -section except for the group which should now be ``ex1a``. +cloning the UM main and meta branches and run the test suite as you did in the +previous section except for the group which should now be ``ex1a``. + +.. note:: + + At the time of writing, the solution for extracting from github on monsoon + is unknown Reset Keywords and Remove Prebuilds (Important!) @@ -229,22 +175,13 @@ Reset Keywords and Remove Prebuilds (Important!) As both yourself and umadmin, -* Remove or comment out the custom keyword revisions from - ``~/.metomi/fcm/keywords.cfg`` * Remove or comment out the ``bypass-version-check`` in ``~/.metomi/rose.conf`` - * Make sure to do this on **all** platforms (inlcuding Monsoon) - * Not doing so can result in some weird behaviour down the line - -* As umadmin, remove the installed release and prebuilds. Doing this now saves - significant time during the actual release. These steps should only need - doing on Azure Spice and EXAB. + * Make sure to do this on **all** platforms (inlcuding Monsoon) + * Not doing so can result in some weird behaviour down the line - * Delete the ``$UMDIR/vnX.Y`` directory - * On Azure Spice, run ``cylc clean --timeout=5000 vnX.Y_prebuilds``. Once - this has finished, check the cylc-run directory that the suite has been - removed. Do this on all of $HOME, $DATADIR, $SCRATCH on both Azure Spice - and EXAB. +* As umadmin, remove the installed release, by deleting the ``$UMDIR/vnX.Y`` + directory The test release is now done! diff --git a/source/Reviewers/releases/updating_prebuilds.rst b/source/Reviewers/releases/updating_prebuilds.rst index e14403a8..066a5eae 100644 --- a/source/Reviewers/releases/updating_prebuilds.rst +++ b/source/Reviewers/releases/updating_prebuilds.rst @@ -1,5 +1,9 @@ .. _updating_prebuilds: +**This page is left in case prebuilds become required again, but at the move to +github, prebuilds are no longer functional, so these instructions are no longer +required** + Mid-Release Prebuilds ===================== diff --git a/source/WorkingPractices/gh_dev_init.rst b/source/WorkingPractices/gh_dev_init.rst index 19644c54..7e1d2c70 100644 --- a/source/WorkingPractices/gh_dev_init.rst +++ b/source/WorkingPractices/gh_dev_init.rst @@ -108,6 +108,7 @@ general development, you should now get a clone of your fork. selecting the url as desired. +.. _create_branch: Create a Branch --------------- diff --git a/source/_static/custom.css b/source/_static/custom.css index e00ce087..2a908a93 100644 --- a/source/_static/custom.css +++ b/source/_static/custom.css @@ -23,4 +23,4 @@ html[data-theme="dark"] { a:visited:hover { color: var(--pst-color-link-hover); } -} \ No newline at end of file +} diff --git a/source/_templates/crown-copyright.html b/source/_templates/crown-copyright.html index b282caab..d0614ad0 100644 --- a/source/_templates/crown-copyright.html +++ b/source/_templates/crown-copyright.html @@ -9,4 +9,4 @@
{% endif %}

-{% endif %} \ No newline at end of file +{% endif %} diff --git a/source/_templates/show-accessibility.html b/source/_templates/show-accessibility.html index fde04744..9a9e0120 100644 --- a/source/_templates/show-accessibility.html +++ b/source/_templates/show-accessibility.html @@ -1,4 +1,4 @@ {# Displays a link to the .rst source of the current page. #}
{{ "Accessibility" }} -
\ No newline at end of file + diff --git a/source/conf.py b/source/conf.py index 7b5628e7..b38c840a 100644 --- a/source/conf.py +++ b/source/conf.py @@ -42,7 +42,7 @@ # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. -exclude_patterns = [] +exclude_patterns = [".venv"] html_static_path = ["_static"] html_css_files = ["custom.css"]