Enable EnKF-only for atmosphere #4345

bhuang95 · 2025-12-16T21:26:08Z

Description

This PR, along with its three dependent PRs listed below, enables running enkf only for atmosphere in the global-workflow with necessary tasks. It will help test JEDI EnKF capabilities in the global workflow and compare with GSI in support of hybrid 4DEnVar appplication for the atmosphere.

This was further modified based on the original branch from Travis J Elless at NOAA EMC.

Major change summary from three PRs:

This PR:

Add DOENKFONLY_ATM (false by default) in the dev/parm/config/gfs/config.base.j2 to activate the GSI and JEDI EnKF-only run.
Modify rocoto task generator files for gfs_cycle to keep a few EnKF-only tasks.
Add two asks: fetchatmanlbias and prepatmanlbias to fetch operational analysis bias correction files at the previous cycle and process them for JEDI and GSI EnKF use (dev/jobs/JGLOBAL_ATM_PREP_ANL_BIAS)

NOAA-EMC/GDASApp#2010:

Rename JEDI increment files generated by enkf-only (without recentering) to be consistent wth ensemble forecast script.
Add a parm/atm/atm_ens_obs_dist_localizations.yaml.j2 to define obs distribution and localization for local_ensemble_da atmposphere applicaton. (see more details in the in the following jcb-gdas and jcb PR.)

NOAA-EMC/jcb-gdas#219:

Add an if statement to use correct JEDI enkf-only increment filetype (without recentering) to be consistent with ensemble forecast script.
Remove observations/atmosphere-lgetkf directory (not used due to changes in jcb below)

NOAA-EMC/jcb#32:

Modify src/jcb/renderer.py to read in GDASApp/parm/atm/atm_ens_obs_dist_localizations.yaml.j2 and add obs distribution and obs localizations blocks to obs configuration in jcb-gdas/ observations/atmosphere/ .
The same obs distribution and obs localizations are applied for all assimilated obs. However, if override_obs_distribution_localizations is defined and override: true in GDASApp/parm/atm/atm_ens_obs_dist_localizations.yaml.j2, these predefined two blocks of matching observations will be overridden. This adds the flexibility to apply different localization methods/lengths for different observations.

Resolves:

Enable EnKF-only for atmosphere #4339

Dependencies:
-NOAA-EMC/GDASApp#2010
-NOAA-EMC/jcb-gdas#219
-NOAA-EMC/jcb#32

Acceptance Criteria

With DOENKFONLY_ATM="NO" by default, it should not affect existing applications in the current global workflow. In other words, it should pass all relevant tests following the general guidelines of building the global workflow.
With DOENKFONLY_ATM="YES" in the config.base during the workflow setup, it should generate related tasks for GSI and JEDI EnKF-only run with additional variable changes in config.base, respectively, and pass all these tasks through rocoto jobs.

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? YES/NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

-Built and cycled successfully on Ursa as follows.

Clone this branch

git clone https://github.com/bhuang95/global-workflow.git
cd global-workflow
git checkout feature/enkf_only_dev
git submodule update --init --recursive

Build the branch following step 7.1.3 here
IC on Ursa
/scratch4/BMC/gsienkf/Bo.Huang/expCodes/Workflow/Data/IC/TravisIC-2022010312/
Set the workflow following steps 7.3.2 and 7.3.3 here. At "7.3.3. Step 3: Check user and experiment settings", additionally define the following variables

DOENKFONLY_ATM="YES" (for both GSI and JEDI run) to turn on EnKF-only run
DOIAU="NO" (for both GSI and JEDI run) or "YES" (for GSI run only, not ready for JEDI yet) to turn on or off IAU
DO_JEDIATMVAR="YES" (for JEDI run only) to run JEDI
DO_JEDIATMENS="YES"(for JEDI run only) to run JEDI

Example of GSI EnKF-only rocoto tasks for the first DA cycle

- gdas_prep                      
- gdas_fetchatmanlbias        
- gdas_prepatmanlbias         
- gdas_prepatmiodaobs         
- enkfgdas_atmensanlinit  
- enkfgdas_atmensanlfv3inc
- enkfgdas_atmensanlfinal 
- enkfgdas_atmensanlobs   
- enkfgdas_atmensanlsol   
- enkfgdas_fcst_mem001-080   
- enkfgdas_epos000-006   
- enkfgdas_esfc               
- enkfgdas_earc_tars_00-01  
- enkfgdas_cleanup

Example of JEDI EnKF-only rocoto tasks for the first DA cycle

- gdas_prep                      
- gdas_fetchatmanlbias        
- gdas_prepatmanlbias         
- enkfgdas_eobs               
- enkfgdas_eupd               
- enkfgdas_ecen000            
- enkfgdas_ediag              
- enkfgdas_fcst_mem001-080 
- enkfgdas_epos000-006   
- enkfgdas_esfc               
- enkfgdas_earc_tars_00-01   
- enkfgdas_cleanup

Checklist

… hard-coded (need to revisit)

…del (global_control.nml.IN)

…e temporary file

TravisElless-NOAA

Thanks for picking up the ball on this @bhuang95. As I'm starting to relearn things, I have a few questions along with some other details I noticed initially.

TravisElless-NOAA · 2025-12-17T15:48:00Z

dev/parm/config/gfs/config.prep.j2

+# If DOENKFONLY_ATM="YES", skip PROCESS_TROPCY
+if [[ "${DOENKFONLY_ATM:-NO}" == "YES" ]] ; then
+   export PROCESS_TROPCY="NO"
+fi


Is this block actually needed? I haven't run with this enkf setting yet, but in default testing PROCESS_TROPCY=NO gets set 3 lines earlier. Does this default change for enkf only?

@TravisElless-NOAA I remember I added this block because PROCESS_TROPCY was set to "YES" somewhere and it failed the task when DOENKFONLY_ATM = "YES". Looks like this block is not needed any more here. I will remove this block and test it in the updated PR.

TravisElless-NOAA · 2025-12-17T16:01:32Z

dev/scripts/exglobal_enkf_earc_tars.py

+            'DOLETKF_OCN', 'IAUFHRS_ENKF', 'NET', 'NMEM_ENS_GFS', 'DO_GSISOILDA', 'DO_LAND_IAU',
+            'DOENKFONLY_ATM']


Cleans up the format of this block better

Suggested change

'DOLETKF_OCN', 'IAUFHRS_ENKF', 'NET', 'NMEM_ENS_GFS', 'DO_GSISOILDA', 'DO_LAND_IAU',

'DOENKFONLY_ATM']

'DOLETKF_OCN', 'IAUFHRS_ENKF', 'NET', 'NMEM_ENS_GFS',

'DO_GSISOILDA', 'DO_LAND_IAU', 'DOENKFONLY_ATM']

@TravisElless-NOAA Will modify as suggested in the updated PR. Thanks.

TravisElless-NOAA · 2025-12-17T16:26:00Z

dev/workflow/applications/gfs_cycled.py


+                # Reset tasks to run enkf-only for atm if do_enkfonly_atm=true
+                if options['do_enkfonly_atm']:
+                    task_names[run] = []


Does this clear out the prep jobs you just added/declared previously (line 389)?

@TravisElless-NOAA Yes. Line 385-390, and Line 429- basically reset task_names[run]. It was meant to only include necessary tasks for enkf-only run and to avoid excessive addition of "if" statements before these lines.

Looking at @aerorahul 's comments suggesting using DO_GSIATMVAR, DO_GSIATMENS, DO_JEDIATMVAR, DO_JEDIATMENS to set up enkf-only run for GSI and JEDI. this may not be preferred and I may need to add if statements instead.

TravisElless-NOAA · 2025-12-17T16:40:44Z

dev/workflow/rocoto/gfs_tasks.py

+            dep_dict = {'type': 'metatask', 'name': 'gdas_fcst', 'offset': f"-{timedelta_to_HMS(self._base['interval_gdas'])}"}
+            deps.append(rocoto.add_dependency(dep_dict))


Did this need to be in your enkfonly if block? Currently this new dependency is being added to the entire suite of workflow options, where it wasn't before. So asking if this was intended or not.

@TravisElless-NOAA Line 143-144 is not necessary to be include in the enkfonly if block. Because the enkfgdas_epmn in the enkfonly if block is dependent on the completion of enkfgdas_fcst on Line 143-144. But we can add them there too.

TravisElless-NOAA · 2025-12-17T17:49:51Z

parm/archive/enkf_grp.yaml.j2

+                    {% if not DOENKFONLY_ATM %}
+                        {% for iaufhr in IAUFHRS %}
+                            {% set iaufhr = iaufhr %}
        - "{{ COMIN_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}recentered_increment.atm.i{{ '%03d' % iaufhr }}.nc"
-                    {% endfor %}  # iaufhr in IAUFHRS
+                        {% endfor %}  # iaufhr in IAUFHRS
+                    {% endif %}


This section will need to be changed to match the JEDI version in lines 27-32. If running GSI version of ENKFONLY will want to archive increment files for that run as well.

@TravisElless-NOAA I see. I will add one block for DOENKFONLY_ATM=YES to match Line 27-29 and add the increment files for GSI enkf-only run as well. Thanks.

DavidHuber-NOAA · 2025-12-17T18:31:14Z

@bhuang95 Can you look through the items that shfmt flagged here and apply the recommended changes?

aerorahul · 2025-12-17T21:05:39Z

@bhuang95
Thank you for this contribution.
I have a couple of high-level suggestions:

We have a variable DO_HYBVAR=YES|NO that determines if we run a hybrid EnVar (DO_HYBVAR=YES) or Variational-only (DO_HYBVAR=NO). In addition to that, with DO_HYBVAR=NO always assumes that one wishes to run the variational solver. You wish to choose to update the prior with an ensemble-only via this PR. I assume you wish to only update the ensemble with JEDI (not the GSI-EnKF). We have a variable DO_JEDIATMENS=YES|NO. Should DO_HYBVAR=NO and DO_JEDIATMENS=YES not be sufficient to update the ensemble without executing the variational solver (GSI or JEDI)? What additional value does DO_ATMENSONLY add that is not covered under DO_HYBVAR=YES AND DO_JEDIATMENS=YES?

	Variational	Ensemble
	DO_GSIATMVAR	DO_GSIATMENS
	DO_JEDIATMVAR	DO_JEDIATMENS

With DO_HYBVAR=YES | NO to choose whether it is a hybrid or not (3DVar|EnKF), and the above variables to choose the appropriate update algorithm.

Testing - for each of the options being added, running the GSI-EnKF only and JEDI-EnKF only, we suggest adding 2 separate tests; e.g. C48_gsiatmensDA.yaml and C48_jediatmensDA.yaml

bhuang95 · 2025-12-18T20:05:22Z

@bhuang95 Thank you for this contribution. I have a couple of high-level suggestions:

We have a variable DO_HYBVAR=YES|NO that determines if we run a hybrid EnVar (DO_HYBVAR=YES) or Variational-only (DO_HYBVAR=NO). In addition to that, with DO_HYBVAR=NO always assumes that one wishes to run the variational solver. You wish to choose to update the prior with an ensemble-only via this PR. I assume you wish to only update the ensemble with JEDI (not the GSI-EnKF). We have a variable DO_JEDIATMENS=YES|NO. Should DO_HYBVAR=NO and DO_JEDIATMENS=YES not be sufficient to update the ensemble without executing the variational solver (GSI or JEDI)? What additional value does DO_ATMENSONLY add that is not covered under DO_HYBVAR=YES AND DO_JEDIATMENS=YES?

Variational Ensemble
DO_GSIATMVAR DO_GSIATMENS
DO_JEDIATMVAR DO_JEDIATMENS
With DO_HYBVAR=YES | NO to choose whether it is a hybrid or not (3DVar|EnKF), and the above variables to choose the appropriate update algorithm.

Testing - for each of the options being added, running the GSI-EnKF only and JEDI-EnKF only, we suggest adding 2 separate tests; e.g. C48_gsiatmensDA.yaml and C48_jediatmensDA.yaml

@aerorahul and @TravisElless-NOAA Thanks for your suggestion. In this PR, we would like to set up enkf-only for both GSI and JEDI so that we can run a comparison of both. I used DO_ATMENSONLY trying to minimize changes to the workflow as it is quite complex. I agree it would be better to use DO_HYBVAR, DO_GSI(JEDI)ATMVAR and DO_GSI(JEDI)ATMENS to achieve this. I will modify this PR as suggested and will look into how to set up the CI tests.

Additionally, I have a another question. The DO_HYBRID run has a lot of tasks (e.g., post-processing tasks). In this PR, do you feel comfortable to only include limited tasks for this enkf-only run as I listed in the description? It would be easier for me to implement.

bhuang95 · 2025-12-18T20:08:40Z

@bhuang95 Can you look through the items that shfmt flagged here and apply the recommended changes?

@DavidHuber-NOAA I did not see obvious error in the shfmt scan log there. Following the information from Gemini, I installed shfmt and ran shfmt dev/job_cards/rocoto/prep.sh > /dev/null in the front terminal, no error message was printed out either. I probably used "shfmt" command incorrectly?

DavidHuber-NOAA · 2025-12-18T20:16:50Z

@bhuang95 by default, shfmt writes the corrections to stdout. You can instead run it in place with the -w flag:

shfmt -w dev/job_cards/rocoto/prep.sh

or if you would like to write a new file so you can do your own comparison, you can use the -f flag:

shfmt dev/job_cards/rocoto/prep.sh -f dev/job_cards/rocoto/prep.sh.shfmt

CoryMartin-NOAA · 2026-01-06T17:59:03Z

@bhuang95 can you start by resolving all the conflicts and then we can go from there on scoping out what is needed for this?

bhuang95 · 2026-01-06T19:08:19Z

@bhuang95 can you start by resolving all the conflicts and then we can go from there on scoping out what is needed for this?

Thanks very much, @CoryMartin-NOAA. I will fix the conflicts including the jcb repo.

This PR works along with the following three dependent PRs to enable the EnKF-only configuration for the atmosphere within the global workflow (see detailed description NOAA-EMC/global-workflow#4345) Dependencies: -NOAA-EMC/global-workflow#4345 -NOAA-EMC/GDASApp#2010 -NOAA-EMC/jcb-gdas#219 Resolve - NOAA-EMC/global-workflow#4339 --------- Co-authored-by: Cory Martin <[email protected]>

bhuang95 · 2026-01-14T01:04:15Z

@bhuang95 can you start by resolving all the conflicts and then we can go from there on scoping out what is needed for this?

@CoryMartin-NOAA I fixed most of shell script failure given what was suggested. I am not sure why the shellcheck still failed and I don't see any obvious issues now (some warning triggered this?). Is there a log file for this shellcheck failure?

aerorahul · 2026-01-14T21:11:22Z

@bhuang95 can you start by resolving all the conflicts and then we can go from there on scoping out what is needed for this?

@CoryMartin-NOAA I fixed most of shell script failure given what was suggested. I am not sure why the shellcheck still failed and I don't see any obvious issues now (some warning triggered this?). Is there a log file for this shellcheck failure?

@WalterKolczynski-NOAA The shellcheck linter workflow is not posting the non-compliant lines on the PR. The workflow also does not show where it encountered an issue. Can we get some help on this please?

aerorahul · 2026-01-14T21:54:38Z

the bash_code_analysis Github action is not working as expected. @WalterKolczynski-NOAA is working on a fix.

bhuang95 · 2026-01-15T20:48:21Z

@CoryMartin-NOAA Finally fixed the shellcheck failure and passed all checks. Learnt a lot how this process works, thank you all!

This PR works along with the following three dependent PRs to enable the EnKF-only configuration for the atmosphere within the global workflow (see detailed description NOAA-EMC/global-workflow#4345) Dependencies: -NOAA-EMC/global-workflow#4345 -NOAA-EMC/GDASApp#2010 -NOAA-EMC/jcb#32 Resolve - NOAA-EMC/global-workflow#4339 --------- Co-authored-by: Cory Martin <[email protected]>

This PR works along with the following three dependent PRs to enable the EnKF-only configuration for the atmosphere within the global workflow (see detailed description NOAA-EMC/global-workflow#4345) Dependencies: -NOAA-EMC/global-workflow#4345 -#2010 -NOAA-EMC/jcb#32 Resolve - NOAA-EMC/global-workflow#4339 --------- Co-authored-by: Cory Martin <[email protected]>

bhuang95 added 20 commits October 31, 2025 22:59

Copy from feature/enkf_only

6db4fe9

GSI and JEDI test passed after EE name change. csg in JEDI file names…

e694072

… hard-coded (need to revisit)

Merge branch 'develop' into feature/enkf_only_dev

86ccf0d

add fetch and prepatmopeanlbc tasks and need to test them

bed26ea

Pass JEDI and GSI test

f32d59c

Cleanup but gsi eobs fauled due to recent create_gsi_info.sh change

b04c54a

Minor change with failure of gsi jobs (create_gsi_info.sh) and UFS mo…

fc6005c

…del (global_control.nml.IN)

Merge branch 'develop' into feature/enkf_only_dev

db790b1

Update submodules

22c0c53

Fix model and gsi oeobs failure and rename satbias to bias

9b2eaa6

Fix python coding norm error

85ed17b

1st clean/ready version that pass JEDI and GSI test

e8806e4

Git shellcheck change to 1st clean version

4f5806d

Git shellcheck change 2 to 1st clean version

be8c479

Address David's comments and need to update to recent NOAA EMC develop

fa9ff54

define letkf obs_distribution_localizations in jcb

7b8065c

remove temporary files

8fd40d6

Issue NOAA-EMC#4339: Enable EnKF-only for atmosphere (1st try)

2c15560

Issue NOAA-EMC#4339: Enable EnKF-only for atmosphere (2nd try): remov…

84731f3

…e temporary file

Issue NOAA-EMC#4339: Enable EnKF-only for atmosphere (3rd try): remov…

6a70bc2

…e temporary file

bhuang95 requested review from DavidHuber-NOAA and aerorahul as code owners December 16, 2025 21:26

aerorahul requested a review from TravisElless-NOAA December 16, 2025 21:30

This was referenced Dec 16, 2025

Enable EnKF-only for atmosphere NOAA-EMC/jcb#32

Merged

Enable EnKF-only for atmosphere NOAA-EMC/jcb-gdas#219

Merged

Enable EnKF-only for atmosphere NOAA-EMC/GDASApp#2010

Open

Enable EnKF-only for atmosphere #4339

Open

TravisElless-NOAA reviewed Dec 17, 2025

View reviewed changes

bhuang95 added 4 commits January 13, 2026 23:56

Merge develop and fix code norm failure

49c8139

Fix shfmt failure (try 1)

e52d039

Fix shfmt failure (try 2)

d2f7005

Fix shfmt failure (try 3)

6d09219

Merge branch 'develop' into feature/enkf_only_dev

7a75ad8

aerorahul mentioned this pull request Jan 14, 2026

Fix bash code analysis issues #4440

Merged

11 tasks

Merge branch 'develop' into feature/enkf_only_dev

b7df0e5

aerorahul and others added 3 commits January 15, 2026 10:21

Merge branch 'develop' into feature/enkf_only_dev

ce5ff04

Fix shfmt failure (try 4)

59755de

Fix shellcheck failure (try 1)

aa45fe9

bhuang95 added 2 commits January 16, 2026 18:35

Merge branch 'develop' into feature/enkf_only_dev

cf2b82b

Update sorc/gdas.cd

95ae053

use parm/jcb-gdas develop in sorc/gdas.cd

26a993d

		'DOLETKF_OCN', 'IAUFHRS_ENKF', 'NET', 'NMEM_ENS_GFS', 'DO_GSISOILDA', 'DO_LAND_IAU',
		'DOENKFONLY_ATM']

		dep_dict = {'type': 'metatask', 'name': 'gdas_fcst', 'offset': f"-{timedelta_to_HMS(self._base['interval_gdas'])}"}
		deps.append(rocoto.add_dependency(dep_dict))

Enable EnKF-only for atmosphere #4345

Are you sure you want to change the base?

Enable EnKF-only for atmosphere #4345

Uh oh!

Conversation

bhuang95 commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Acceptance Criteria

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

TravisElless-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA commented Dec 17, 2025

Uh oh!

aerorahul commented Dec 17, 2025

Uh oh!

bhuang95 commented Dec 18, 2025

Uh oh!

bhuang95 commented Dec 18, 2025

Uh oh!

DavidHuber-NOAA commented Dec 18, 2025

Uh oh!

CoryMartin-NOAA commented Jan 6, 2026

Uh oh!

bhuang95 commented Jan 6, 2026

Uh oh!

bhuang95 commented Jan 14, 2026

Uh oh!

aerorahul commented Jan 14, 2026

Uh oh!

aerorahul commented Jan 14, 2026

Uh oh!

bhuang95 commented Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bhuang95 commented Dec 16, 2025 •

edited

Loading