Enable EnKF-only for atmosphere #2010

bhuang95 · 2025-12-16T21:24:10Z

This PR works along with the following three dependent PRs to enable the EnKF-only configuration for the atmosphere within the global workflow (see detailed description NOAA-EMC/global-workflow#4345)

Dependencies:
-NOAA-EMC/global-workflow#4345
-NOAA-EMC/jcb-gdas#219
-NOAA-EMC/jcb#32

Resolve

Enable EnKF-only for atmosphere global-workflow#4339

… hard-coded (need to revisit)

parm/atm/atm_ens_obs_dist_localizations.yaml.j2

This PR works along with the following three dependent PRs to enable the EnKF-only configuration for the atmosphere within the global workflow (see detailed description NOAA-EMC/global-workflow#4345) Dependencies: -NOAA-EMC/global-workflow#4345 -NOAA-EMC/GDASApp#2010 -NOAA-EMC/jcb-gdas#219 Resolve - NOAA-EMC/global-workflow#4339 --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

bhuang95 · 2026-01-09T23:14:16Z

@CoryMartin-NOAA I updated this branch with develop. It now has issues of reading the obs distribution and localization blocks in atm_ens_obs_dist_localizations.yaml.j2
to configure LETKF yaml wtih below errors. This yaml is used on Line 23 of this LETKF YAML template. Do you know what recent changes may cause this? Looks like something is wrong with the format in this yaml?

1251 ^[[38;5;39m2026-01-09 22:53:31,307 - DEBUG    - jedi        : Writing JEDI YAML config to: /scratch4/BMC/gsienkf/Bo.Huang/expCodes/Workflow/EnKFOnly-20251031/TestGWSet     upRealCase/tmp/RUNDIRS/TestGW-EnKFOnly-JEDI-MEM3-T11/enkfgdas.2022010318/enkfgdasatmensanl_18/atmensanlobs.yaml^[[0m
1252 Traceback (most recent call last):
1253   File "/scratch4/BMC/gsienkf/Bo.Huang/expCodes/Workflow/EnKFOnly-20251031/global-workflow/scripts/exglobal_atmens_analysis_initialize.py", line 26, in <module>
1254     AtmEnsAnl.initialize()
1255   File "/scratch4/BMC/gsienkf/Bo.Huang/expCodes/Workflow/EnKFOnly-20251031/global-workflow/sorc/wxflow/src/wxflow/logger.py", line 252, in wrapper
1256     retval = func(*args, **kwargs)
1257              ^^^^^^^^^^^^^^^^^^^^^
1258   File "/scratch4/BMC/gsienkf/Bo.Huang/expCodes/Workflow/EnKFOnly-20251031/global-workflow/ush/python/pygfs/task/atmens_analysis.py", line 90, in initialize
1259     self.jedi_dict['atmensanlobs'].initialize(clean_empty_obsspaces=True)
1260   File "/scratch4/BMC/gsienkf/Bo.Huang/expCodes/Workflow/EnKFOnly-20251031/global-workflow/sorc/wxflow/src/wxflow/logger.py", line 252, in wrapper
1261     retval = func(*args, **kwargs)
1262              ^^^^^^^^^^^^^^^^^^^^^
1263   File "/scratch4/BMC/gsienkf/Bo.Huang/expCodes/Workflow/EnKFOnly-20251031/global-workflow/ush/python/pygfs/jedi/jedi.py", line 189, in initialize
1264     save_as_yaml(self.exe_config, self.jedi_config.exe_config_yaml)
1265   File "/scratch4/BMC/gsienkf/Bo.Huang/expCodes/Workflow/EnKFOnly-20251031/global-workflow/sorc/wxflow/src/wxflow/yaml_file.py", line 54, in save_as_yaml
1266     yaml.safe_dump(vanilla_yaml(data), fh,
...

1320   File "/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.2.1/install/oneapi/2024.2.1/py-pyyaml-6.0.2-55i3htn/lib/python3.11/site-packages/yaml/representer.py     ", line 58, in represent_data
1321     node = self.yaml_representers[None](self, data)
1322            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1323   File "/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.2.1/install/oneapi/2024.2.1/py-pyyaml-6.0.2-55i3htn/lib/python3.11/site-packages/yaml/representer.py     ", line 231, in represent_undefined
1324     raise RepresenterError("cannot represent an object", data)
1325 yaml.representer.RepresenterError: ('cannot represent an object', {'name': 'RoundRobin', 'halo size': '1250e3'})

CoryMartin-NOAA · 2026-01-12T14:25:36Z

@bhuang95 I ran into this same problem on Thursday. I don't immediately know what new change broke this, but I think the fix is this: NOAA-EMC/wxflow#65

This PR works along with the following three dependent PRs to enable the EnKF-only configuration for the atmosphere within the global workflow (see detailed description NOAA-EMC/global-workflow#4345) Dependencies: -NOAA-EMC/global-workflow#4345 -NOAA-EMC/GDASApp#2010 -NOAA-EMC/jcb#32 Resolve - NOAA-EMC/global-workflow#4339 --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

This PR works along with the following three dependent PRs to enable the EnKF-only configuration for the atmosphere within the global workflow (see detailed description NOAA-EMC/global-workflow#4345) Dependencies: -NOAA-EMC/global-workflow#4345 -#2010 -NOAA-EMC/jcb#32 Resolve - NOAA-EMC/global-workflow#4339 --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

CoryMartin-NOAA · 2026-01-22T18:17:13Z

@bhuang95 does any change here depend on global-workflow or is it the other way around? Meaning, can we (after you or I resolve conflicts) work towards getting this merged in ASAP?

…bal-workflow

bhuang95 · 2026-01-22T19:50:59Z

@bhuang95 does any change here depend on global-workflow or is it the other way around? Meaning, can we (after you or I resolve conflicts) work towards getting this merged in ASAP?

@CoryMartin-NOAA These two lines with default value: false below should remove the dependency of this PR on the global workflow, and the other way around.

GDASApp/parm/atm/atm_ens_config.yaml.j2

Line 78 in 9b75e70

{% if DOENKFONLY_ATM | default(false) %}

GDASApp/parm/atm/jcb-base.yaml.j2

Line 132 in 9b75e70

atm_enkfonly: {{ DOENKFONLY_ATM | default(false, true) }}

The other major changes about obs distribution and localization will be handled in the NOAA-EMC/jcb#32 (merged) and use atmosphere (without obs localization setting) directory instead of atmosphere-lgetkf https://github.com/NOAA-EMC/GDASApp/tree/develop/parm/jcb-gdas/observations. I think both will not affect the upstream and downstream either.

I addressed the conflicts. I think it should be ready to merge if it looks good to you.

CoryMartin-NOAA

Looks good, thanks @bhuang95 I'm going to test this and then approve if it works well

parm/atm/jcb-prototype_lgetkf.yaml.j2

CoryMartin-NOAA

Approved but we need to not merge until the workflow PR is ready or else things will be broken

CoryMartin-NOAA

Tested manually on WCOSS2, and confirmed that it creates a YAML for the observer that looks good. Thanks, this will really help with YAML maintenance going forward.

bhuang95 added 16 commits October 22, 2025 16:41

Use atmosphere obs directory for lgetkf

61100eb

Initial pass of GSI and JEDI, but forecast failed with NaN

5345992

Add atm_letkf_app

9f22d92

Fix conflicts when merging develop

15ed146

Changes for EE name standards

f241807

GSI and JEDI test passed after EE name change. csg in JEDI file names…

30baa19

… hard-coded (need to revisit)

add fetch and prepatmopeanlbc tasks and need to test them

299f38f

Pass GSI and JEDI test

f3131ae

Cleanup but gsi eobs fauled due to recent create_gsi_info.sh change

dc3b930

Merge branch 'develop' into feature/enkf_only_dev

50088cb

Update submodules

8cb610e

1st clean/ready version that pass JEDI and GSI test

c0651a3

Address David's comments and need to update to recent NOAA EMC develop

3cc019d

define letkf obs_distribution_localizations in jcb

c3b2bab

remove temporary files

2e104c7

Issue #4339: Enable EnKF-only for atmosphere (1st try)

e3cc5c9

bhuang95 changed the title ~~Feature/enkf only dev~~ Enable EnKF-only for atmosphere Dec 16, 2025

This was referenced Dec 16, 2025

Enable EnKF-only for atmosphere NOAA-EMC/global-workflow#4345

Open

Enable EnKF-only for atmosphere NOAA-EMC/jcb#32

Merged

Enable EnKF-only for atmosphere NOAA-EMC/jcb-gdas#219

Merged

CoryMartin-NOAA reviewed Jan 2, 2026

View reviewed changes

parm/atm/atm_ens_obs_dist_localizations.yaml.j2 Show resolved Hide resolved

bhuang95 added 4 commits January 13, 2026 23:52

Merge develop and fix code norm failure

882c00f

Merge branch 'develop' into feature/enkf_only_dev

18d173f

Update sorc/jcb to develop and parm/jcb-gdas

14387b0

Update parm/jcb-gdas

78ab701

use parm/jcb-gdas develop

bfb20db

Add default(false) for DOENKFONLY_ATM to remove its dependency on glo…

9b75e70

…bal-workflow

CoryMartin-NOAA reviewed Jan 22, 2026

View reviewed changes

parm/atm/jcb-prototype_lgetkf.yaml.j2 Outdated Show resolved Hide resolved

CoryMartin-NOAA previously approved these changes Jan 22, 2026

View reviewed changes

Directly include atm_ens_obs_dist_localizations.yaml.j2

05fbd82

bhuang95 dismissed CoryMartin-NOAA’s stale review via 05fbd82 January 22, 2026 22:18

Correct fr lgetkf

e646e79

DavidNew-NOAA approved these changes Jan 23, 2026

View reviewed changes

CoryMartin-NOAA approved these changes Jan 23, 2026

View reviewed changes

CoryMartin-NOAA merged commit 9fe6905 into NOAA-EMC:develop Jan 23, 2026
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable EnKF-only for atmosphere #2010

Enable EnKF-only for atmosphere #2010

bhuang95 commented Dec 16, 2025 •

edited

Loading

Uh oh!

Uh oh!

bhuang95 commented Jan 9, 2026 •

edited

Loading

Uh oh!

CoryMartin-NOAA commented Jan 12, 2026

Uh oh!

CoryMartin-NOAA commented Jan 22, 2026

Uh oh!

bhuang95 commented Jan 22, 2026

Uh oh!

CoryMartin-NOAA left a comment

Uh oh!

Uh oh!

CoryMartin-NOAA left a comment

Uh oh!

CoryMartin-NOAA left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Enable EnKF-only for atmosphere #2010

Enable EnKF-only for atmosphere #2010

Conversation

bhuang95 commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

bhuang95 commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CoryMartin-NOAA commented Jan 12, 2026

Uh oh!

CoryMartin-NOAA commented Jan 22, 2026

Uh oh!

bhuang95 commented Jan 22, 2026

Uh oh!

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bhuang95 commented Dec 16, 2025 •

edited

Loading

bhuang95 commented Jan 9, 2026 •

edited

Loading