Preprocess and assimilate the GHCN snow data #4386

jiaruidong2017 · 2025-12-27T01:14:02Z

Description

This PR adds an entry to enable assimilation of the GHCN snow observations for reanalysis. This PR also includes functionality to preprocess the GHCN snow observations for global snow analysis.

This PR depends NOAA-EMC/GDASApp#2019; #4388; NOAA-EMC/GDASApp#2027
Resolves NOAA-EMC/GDASApp#2018

This PR merged the PR #4387 which enables to read and process the input IMS snow cover data in either ascii or netCDF format.
Resolves NOAA-EMC/GDASApp#2022

Type of change

New feature (adds functionality)

Change characteristics

Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? NO
Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? YES

How has this been tested?

I conducted cycled experiments with DO_JEDISNOWDA=YES. These runs also require GHCN snow observations.

Cycled test on Ursa

Checklist

ClaraDraper-NOAA

For both IMS and GHCN, the snow_analysis task will continue without failing if there's a problem with the input data.

If there's a problem in prepare_[dataset], it reports a warning and continues. The snow_analysis will also continue if it doesn't find the expected file. I'd prefer that the code exits (or perhaps has a config option to force exit?) to let us know that there was a problem with the file. @CoryMartin-NOAA ?

sorc/link_workflow.sh

ush/python/pygfs/task/snow_analysis.py

CoryMartin-NOAA · 2025-12-30T14:29:49Z

@ClaraDraper-NOAA we can make an option that will fail instead of warn, but except in rare circumstances, operational requirements will make it so that the code needs to run to completion if input data is missing. Assimilating no snow obs that cycle is 10000x preferred to the GFS not starting in a timely manner.

ClaraDraper-NOAA · 2025-12-30T14:47:52Z

@ClaraDraper-NOAA we can make an option that will fail instead of warn, but except in rare circumstances, operational requirements will make it so that the code needs to run to completion if input data is missing. Assimilating no snow obs that cycle is 10000x preferred to the GFS not starting in a timely manner.

thanks @CoryMartin-NOAA . @jiaruidong2017 Can you please make an option for GHCN and IMS to switch between warning and failing if the obs are missing. It should be controlled by an input flag, with the default being to throw a warning.

jiaruidong2017 · 2025-12-31T02:47:43Z

@ClaraDraper-NOAA we can make an option that will fail instead of warn, but except in rare circumstances, operational requirements will make it so that the code needs to run to completion if input data is missing. Assimilating no snow obs that cycle is 10000x preferred to the GFS not starting in a timely manner.

thanks @CoryMartin-NOAA . @jiaruidong2017 Can you please make an option for GHCN and IMS to switch between warning and failing if the obs are missing. It should be controlled by an input flag, with the default being to throw a warning.

If the GHCN data is missing, the GHCN preprocessing (prepare_GHCN()) will not run as shown below.

        # if 00z, do GHCN preprocessing
        _ghcn_file = os.path.join(self.task_config.COMIN_OBS, f'{self.task_config.OPREFIX}ghcn_snow.csv')
        logger.info(f"Checking for GHCN csv file: {_ghcn_file}")
        if self.task_config.cyc == 0 and os.path.exists(_ghcn_file):
            _DO_GHCN = True
        else:
            _DO_GHCN = False

Therefore, the warning in prepare_GHCN() is unnecessary because this condition will never occur in practice. It can be safely removed. What do you think?

        if not os.path.isfile(csv_file):
            logger.warning(f"WARNING: GHCN obs file not found.")
            return

If we still decide to keep the logic that allows the code to exit when the GHCN data is missing, then this check should be implemented during the initialize() step as shown below:

        # if 00z, do GHCN preprocessing
        _DO_GHCN = False
        if self.task_config.cyc == 0:
            if os.path.exists(_ghcn_file):
                _DO_GHCN = True
            else:
                if _fail_on_missing:
                    raise FileNotFoundError(
                        f"GHCN obs file required but not found: {_ghcn_file}"
                    )
                else:
                    logger.warning(f"GHCN obs file missing: {_ghcn_file}")
        else:
            logger.info("Not 00z cycle — Skipping GHCN preprocessing.")

Any suggestions?

CoryMartin-NOAA · 2025-12-31T13:12:38Z

@jiaruidong2017 your last snippet makes sense to me, raise a FileNotFound error if the file is missing and we want it to not continue when missing

ClaraDraper-NOAA · 2025-12-31T14:19:24Z

Thanks @jiaruidong2017. I like your last solution too. can you please add it for IMS as well?

jiaruidong2017 · 2025-12-31T14:23:08Z

Thanks @CoryMartin-NOAA @ClaraDraper-NOAA I will add it for both GHCN and IMS.

dev/parm/config/gfs/config.snowanl.j2

# Description This PR adds a `ghcn_snod2ioda.py` script to processes the GHCN data in csv format to IODA-format files for snow DA in the global-workflow. # Companion PRs This PR depends NOAA-EMC/jcb-gdas#220, NOAA-EMC/DA-utils#53, and NOAA-EMC/global-workflow#4388 This PR contributes NOAA-EMC/global-workflow#4386 # Issues Resolves #2018 # Automated CI tests to run in Global Workflow  - [ ] atm_jjob  - [ ] C96C48_ufs_hybatmDA  - [ ] C96C48_hybatmsnowDA  - [ ] C96_gcafs_cycled  - [ ] C48mx500_3DVarAOWCDA  - [ ] C48mx500_hybAOWCDA  - [ ] C96C48_ufsgsi_hybatmDA  - [ ] C96C48_hybatmDA  --------- Co-authored-by: Cory Martin <[email protected]>

jiaruidong2017 · 2026-01-05T14:45:15Z

Thank you @RussTreadon-NOAA As you suggested, I update the config.esnowanl.j2 to include the required parameters settings for snow ensemble analysis.

DavidHuber-NOAA · 2026-01-06T18:04:34Z

Launching CI on C6.

emcbot · 2026-01-06T19:56:38Z

C96C48_hybatmsnowDA FAILED on Gaeac6 (pipeline ID: 6829)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/EXPDIR/C96C48_hybatmsnowDA_bc6c2927-6829

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/COMROOT/C96C48_hybatmsnowDA_bc6c2927-6829/logs/2021122018/gdas_anlstat.log

View Error Logs: (gdas_anlstat.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2026-01-06T19:56:41Z

C48mx500_hybAOWCDA FAILED on Gaeac6 (pipeline ID: 6829)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_bc6c2927-6829

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_bc6c2927-6829/logs/2021032500/gdas_anlstat.log

View Error Logs: (gdas_anlstat.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2026-01-06T19:56:44Z

C96C48_hybatmDA FAILED on Gaeac6 (pipeline ID: 6829)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/EXPDIR/C96C48_hybatmDA_bc6c2927-6829

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/COMROOT/C96C48_hybatmDA_bc6c2927-6829/logs/2021122100/gdas_anlstat.log

View Error Logs: (gdas_anlstat.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2026-01-06T19:56:53Z

C96_atm3DVar FAILED on Gaeac6 (pipeline ID: 6829)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/EXPDIR/C96_atm3DVar_bc6c2927-6829

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/COMROOT/C96_atm3DVar_bc6c2927-6829/logs/2021122100/gdas_anlstat.log

View Error Logs: (gdas_anlstat.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2026-01-06T19:56:55Z

C48mx500_3DVarAOWCDA FAILED on Gaeac6 (pipeline ID: 6829)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_bc6c2927-6829

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_bc6c2927-6829/logs/2021032500/gdas_anlstat.log

View Error Logs: (gdas_anlstat.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

CoryMartin-NOAA · 2026-01-06T19:59:33Z

@DavidHuber-NOAA I think we are going to have to just accept that this PR has broken stats and the other has broken snow and merge both in and I will fix ASAP any issues that were missed. Apologies but it's from juggling too many pieces in the syncing of PRs between repos.

jiaruidong2017 · 2026-01-06T20:00:41Z

I think the gdas_anlstat issue can be fixed by @CoryMartin-NOAA's PR #4401

jiaruidong2017 · 2026-01-06T20:01:15Z

Thanks @CoryMartin-NOAA

emcbot · 2026-01-06T20:08:51Z

C96_gcafs_cycled FAILED on Gaeac6 (pipeline ID: 6829)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/EXPDIR/C96_gcafs_cycled_bc6c2927-6829

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/COMROOT/C96_gcafs_cycled_bc6c2927-6829/logs/2021122018/gcdas_anlstat.log

View Error Logs: (gcdas_anlstat.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2026-01-06T20:16:06Z

C96C48mx500_S2SW_cyc_gfs FAILED on Gaeac6 (pipeline ID: 6829)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_bc6c2927-6829

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_bc6c2927-6829/logs/2021122018/gdas_anlstat.log

View Error Logs: (gdas_anlstat.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2026-01-06T20:52:43Z

C96C48_hybatmsoilDA FAILED on Gaeac6 (pipeline ID: 6829)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/EXPDIR/C96C48_hybatmsoilDA_bc6c2927-6829

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4386_bc6c2927_6829/RUNTESTS/COMROOT/C96C48_hybatmsoilDA_bc6c2927-6829/logs/2022051512/gdas_anlstat.log

View Error Logs: (gdas_anlstat.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

DavidHuber-NOAA · 2026-01-07T14:32:19Z

Closing as this has been merged into #4401.

# Description This PR adds a `ghcn_snod2ioda.py` script to processes the GHCN data in csv format to IODA-format files for snow DA in the global-workflow. # Companion PRs This PR depends NOAA-EMC/jcb-gdas#220, NOAA-EMC/DA-utils#53, and NOAA-EMC/global-workflow#4388 This PR contributes NOAA-EMC/global-workflow#4386 # Issues Resolves #2018 # Automated CI tests to run in Global Workflow  - [ ] atm_jjob  - [ ] C96C48_ufs_hybatmDA  - [ ] C96C48_hybatmsnowDA  - [ ] C96_gcafs_cycled  - [ ] C48mx500_3DVarAOWCDA  - [ ] C48mx500_hybAOWCDA  - [ ] C96C48_ufsgsi_hybatmDA  - [ ] C96C48_hybatmDA  --------- Co-authored-by: Cory Martin <[email protected]>

# Description This PR updates the `da-utils` commit hash to include the required utilities. This PR also addresses Copilot comments in `ghcn_snod2ioda.py`. In particular, it fixes the use of AttrData and DimDict as mutable module-level dictionaries, which is error-prone—especially since DimDict is modified within the class. These structures are now handled in a safer, more maintainable way. This PR is a supplementary update to PR #2019 and contributes to NOAA-EMC/global-workflow#4386 # Issues Resolves #2018 # Automated CI tests to run in Global Workflow  - [ ] atm_jjob  - [ ] C96C48_ufs_hybatmDA  - [ ] C96C48_hybatmsnowDA  - [ ] C96_gcafs_cycled  - [ ] C48mx500_3DVarAOWCDA  - [ ] C48mx500_hybAOWCDA  - [ ] C96C48_ufsgsi_hybatmDA  - [ ] C96C48_hybatmDA

Preprocess and assimilate the GHCN snow data

d1d6612

jiaruidong2017 requested review from DavidHuber-NOAA, DavidNew-NOAA and aerorahul as code owners December 27, 2025 01:14

jiaruidong2017 marked this pull request as draft December 27, 2025 01:14

jiaruidong2017 self-assigned this Dec 27, 2025

jiaruidong2017 added the JEDI Feature development to support JEDI-based DA label Dec 27, 2025

jiaruidong2017 requested review from ClaraDraper-NOAA and CoryMartin-NOAA December 27, 2025 01:15

jiaruidong2017 mentioned this pull request Dec 27, 2025

Add scripts to preprocess the GHCN snow data NOAA-EMC/GDASApp#2019

Merged

8 tasks

Add GHCN preprocessing into the snow ensemble analysis

805ee0a

jiaruidong2017 mentioned this pull request Dec 29, 2025

Add GHCN station index file for snow DA #4388

Closed

59 tasks

ClaraDraper-NOAA reviewed Dec 29, 2025

View reviewed changes

sorc/link_workflow.sh Show resolved Hide resolved

ush/python/pygfs/task/snow_analysis.py Show resolved Hide resolved

jiaruidong2017 mentioned this pull request Dec 30, 2025

Add GHCN assimilation to global_workflow for use in the reanalysis ClaraDraper-NOAA/task_tracker#35

Closed

jiaruidong2017 added 2 commits December 31, 2025 09:48

Add an option for manually setting a run to fail or warn

1cda206

Merge branch 'develop' into feature/snow-ghcn2

3933d6e

jiaruidong2017 mentioned this pull request Jan 2, 2026

Enable to read IMS data in ascii or netcdf format #4387

Closed

12 tasks

Merge PR NOAA-EMC#4387 into this PR

f2c6847

jiaruidong2017 mentioned this pull request Jan 2, 2026

Work to enable IMS preprocessing support for netCDF input files ClaraDraper-NOAA/task_tracker#36

Closed

CoryMartin-NOAA reviewed Jan 2, 2026

View reviewed changes

dev/parm/config/gfs/config.snowanl.j2 Outdated Show resolved Hide resolved

RussTreadon-NOAA linked an issue Jan 2, 2026 that may be closed by this pull request

Remove gdas_calcfIMS.x from link_workflow.sh when updating GDASApp submodule #4398

Open

Rename the environment variabe to be more specific

a52892f

jiaruidong2017 mentioned this pull request Jan 2, 2026

Update da-utils commit hash and address the co-pilot comments NOAA-EMC/GDASApp#2027

Merged

8 tasks

Update config.esnowanl.j2 to include the required parameters

82a104a

This was referenced Jan 5, 2026

anlstat jobs fail in multiple g-w CI cases #4400

Open

Include fixes to properly handle GSI ncdiag files in anlstat job #4401

Merged

Add DO_GHCN in the script for snow ensemble analysis

f374702

CoryMartin-NOAA approved these changes Jan 6, 2026

View reviewed changes

DavidHuber-NOAA approved these changes Jan 6, 2026

View reviewed changes

emcbot added CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed and removed CI-Gaeac6-Running **Bot use only** CI testing on Gaea C6 for this PR is in-progress labels Jan 6, 2026

DavidHuber-NOAA closed this Jan 7, 2026

Preprocess and assimilate the GHCN snow data #4386

Preprocess and assimilate the GHCN snow data #4386

Uh oh!

Conversation

jiaruidong2017 commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

ClaraDraper-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

CoryMartin-NOAA commented Dec 30, 2025

Uh oh!

ClaraDraper-NOAA commented Dec 30, 2025

Uh oh!

jiaruidong2017 commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CoryMartin-NOAA commented Dec 31, 2025

Uh oh!

ClaraDraper-NOAA commented Dec 31, 2025

Uh oh!

jiaruidong2017 commented Dec 31, 2025

Uh oh!

Uh oh!

jiaruidong2017 commented Jan 5, 2026

Uh oh!

DavidHuber-NOAA commented Jan 6, 2026

Uh oh!

emcbot commented Jan 6, 2026

Uh oh!

emcbot commented Jan 6, 2026

Uh oh!

emcbot commented Jan 6, 2026

Uh oh!

emcbot commented Jan 6, 2026

Uh oh!

emcbot commented Jan 6, 2026

Uh oh!

CoryMartin-NOAA commented Jan 6, 2026

Uh oh!

jiaruidong2017 commented Jan 6, 2026

Uh oh!

jiaruidong2017 commented Jan 6, 2026

Uh oh!

emcbot commented Jan 6, 2026

Uh oh!

emcbot commented Jan 6, 2026

Uh oh!

emcbot commented Jan 6, 2026

Uh oh!

DavidHuber-NOAA commented Jan 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jiaruidong2017 commented Dec 27, 2025 •

edited

Loading

jiaruidong2017 commented Dec 31, 2025 •

edited

Loading