Enable coupling of EnKF and Var DA cycling experiments

[st-ncar]

Description

This PR introduces the capability to couple concurrently running Var DA and EnKF cycling experiments. Each experiment runs its own Cylc flow and the inter-dependency is introduced via an external workflow_state trigger (see https://cylc.github.io/cylc-doc/8.1.4/html/user-guide/writing-workflows/external-triggers.html for details). Coupled experiments exchange the following data (a -> b means "a is coupled to b"):

1. Var DA -> EnKF: Var DA uses the EnKF forecast ensemble to construct the ensemble B. New external dependency: the Var DA analysis at step n can start once the EnKF forecast at step n-1 has finished.
2. EnKF -> Var DA: EnKF uses the bias correction files from Var DA and/or the EnKF analysis ensemble is recentered on the Var DA analysis. New external dependency: the EnKF analysis at step n can start once the Var DA analysis at step n has finished.
3. Var DA <-> EnKF: 1 + 2. This is the typical use case.

The coupling via external triggers is based on the workflow ID, which is generated from the workflow name. It is recommended to specify the prefix, name, and suffix of the experiment in the yaml files to make sure the coupled workflows can be identified correctly.

The two new scenarios, getkf_OIE120km_coupled.yaml and 3denvar_OIE120km_coupled.yaml demonstrate how to run coupled experiments.

Note: the recentering application can interpolate between different EnKF and Var DA resolutions. But the current approach is not yet tested may not lead to good forecast accuracy.

Issue closed

N/A.

Tests completed

I confirmed that the external triggers behave as expected and did model space verification for a number of uncoupled and coupled data assimilation runs. I will update this section as the PR matures.

Tier 1:

• 3dvar_OIE120km_WarmStart
• 3denvar_OIE120km_IAU_WarmStart
• 3dvar_OIE120km_ColdStart
• 3dvar_O30kmIE60km_ColdStart
• 3denvar_O30kmIE60km_WarmStart
• eda_OIE120km_WarmStart
• getkf_OIE120km_WarmStart
• ForecastFromGFSAnalysesMPT
• 4denvar_OIE120km_WarmStart
• 4dhybrid_OIE120km_WarmStart

Tier 2 (optional):

• GenerateGFSAnalyses
• GenerateObs