Refactor Tar Archiving Logic to Python-Based Implementation #4336

AntonMFernando-NOAA · 2025-12-15T16:00:40Z

Description

This PR refactors the tar archiving (arch_tar) workflow to move all logic, cycle variable handling, and COM path resolution from YAML/Jinja templates into Python. YAML configuration is now limited to describing what files are archived, while Python scripts are responsible for how variables are generated and resolved.
This change aligns tar archiving with the local archiving (arch_vrfy) approach and establishes a unified, Python-driven archive framework.

Resolves [Enabler] Review the archiving process and determine best paths forward #3656
Refs Archive Enhancement — Migrate Template Variables from Jinja to Python #3910

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? YES/NO (If YES, please indicate to which system(s))
- GFS
- GEFS
- SFS
- GCAFS
Is this a breaking change (a change in existing functionality)?NO
Does this change require a documentation update? /NO
Does this change require an update to any of the following submodules? NO
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

Alll available CI tests on Hera and extended tests on WCOSS2

Checklist

…do-NOAA/global-workflow into enabler/archive_tar

ush/python/pygfs/utils/archive_tar_vars.py

…do-NOAA/global-workflow into enabler/archive_tar

ush/python/pygfs/task/archive_tar_vars.py

DavidHuber-NOAA · 2026-01-14T17:27:43Z

Launching CI on C6.

emcbot · 2026-01-14T18:02:47Z

BUILD FAILED on Gaeac6 (pipeline ID: 7151)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4336_f6105b9e_7151/global-workflow/sorc

Error log files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4336_f6105b9e_7151/global-workflow/sorc/logs/gsi_enkf.log

View Error Logs

This failure was detected automatically by global-workflow's CI/CD Pipeline

DavidHuber-NOAA · 2026-01-14T18:10:33Z

Building failed on C6. This issue has been reported to GFDL already. I will pick this testing up manually on C6. I am also launching CI on Dogwood.

DavidHuber-NOAA

I performed an extended C96C48mx500_S2SW_cyc_gfs case on WCOSS2 with all downstream jobs enabled (gempak, awips, goes, etc) in both this branch and develop for 5 cycles and then extracted all tarballs and compared contents. All files were present except the following from the last two cycles (2021122118 and 2021122200):

2021122118/gdas.20211221/18/products/atmos/oznmon/time/bad_pen.2021122118
2021122118/gfs.20211221/18/products/atmos/cyclone/genesis_vital/storms.gfso.atcf_gen.2021122118
2021122118/gfs.20211221/18/products/atmos/cyclone/genesis_vital/trak.gfso.atcfunix.2021122118
2021122200/gdas.20211222/00/products/atmos/oznmon/time/bad_pen.2021122200

I am not certain which tarballs these are located in.

EDIT: I removed the XML because it does actually exist.

DavidHuber-NOAA · 2026-01-16T15:57:34Z

@AntonMFernando-NOAA Can you please take a look at the tarballs in /lfs/h2/emc/ptmp/david.huber/atardir/C96C48mx500_S2SW_cyc_gfs_baseline and check which tarballs these are located in then add them in the appropriate location in your branch?

DavidHuber-NOAA · 2026-01-20T20:33:02Z

After some investigation, I see now that the cyclone files were not being generated in the extended case ran on this branch because the forecast length was only 120 hours. Looking at another case that @AntonMFernando-NOAA ran on Hera, these files were generated and archived successfully.

DavidHuber-NOAA · 2026-01-20T20:40:58Z

Additionally, my tests to find missing files had a flaw in it that incorrectly identified the bad_pen.2021122118 file as missing. It is in fact present.

Thus, all files are accounted for. All tests have passed and this PR is ready to merge.

AntonMFernando-NOAA and others added 15 commits December 9, 2025 14:53

test

a61ed7c

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

edaa085

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

31b6fc0

rename vars

6703cd0

Merge branch 'enabler/archive_tar' of https://github.com/AntonMFernan…

1dbc3dd

…do-NOAA/global-workflow into enabler/archive_tar

remove com paths from master_enkf.yaml

faf93e8

remove deepcopy

fff50be

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

461e362

cyc_vars migrations from sub-yamls

5497488

update parm/archive/master_enkf.yaml.j2

0857700

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

4776694

change yaml vars

a8c4b97

Merge branch 'enabler/archive_tar' of https://github.com/AntonMFernan…

05d4e21

…do-NOAA/global-workflow into enabler/archive_tar

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

63de9a4

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

04bcc71

AntonMFernando-NOAA self-assigned this Dec 15, 2025

aerorahul reviewed Dec 15, 2025

View reviewed changes

ush/python/pygfs/utils/archive_tar_vars.py Outdated Show resolved Hide resolved

ush/python/pygfs/utils/archive_tar_vars.py Show resolved Hide resolved

ush/python/pygfs/utils/archive_tar_vars.py Outdated Show resolved Hide resolved

AntonMFernando-NOAA and others added 3 commits December 15, 2025 13:48

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

bd6f4f1

removed _get_cyc_vars() method

19c912f

Merge branch 'enabler/archive_tar' of https://github.com/AntonMFernan…

010183b

…do-NOAA/global-workflow into enabler/archive_tar

aerorahul reviewed Dec 15, 2025

View reviewed changes

ush/python/pygfs/task/archive_tar_vars.py Outdated Show resolved Hide resolved

removed a duplicate test file

847ca33

AntonMFernando-NOAA requested a review from DavidHuber-NOAA December 15, 2025 19:16

AntonMFernando-NOAA and others added 7 commits December 15, 2025 19:33

simplification

fef813a

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

3d047a0

update archive scripts

a6e39ce

update the archive.py

6c2a48d

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

2e6dcdb

Delete verify_refactor.py

fc45122

added more cyc vars

e428f7c

DavidHuber-NOAA approved these changes Jan 14, 2026

View reviewed changes

emcbot added CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 and removed CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 labels Jan 14, 2026

emcbot added CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed and removed CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 labels Jan 14, 2026

AntonMFernando-NOAA added 3 commits January 14, 2026 16:50

Merge branch 'develop' into enabler/archive_tar

089e78a

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

df447f9

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

59bde5d

DavidHuber-NOAA added CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress labels Jan 16, 2026

DavidHuber-NOAA requested changes Jan 16, 2026

View reviewed changes

AntonMFernando-NOAA added 3 commits January 16, 2026 15:13

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

796bf51

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

f255437

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

53cdd0d

DavidHuber-NOAA approved these changes Jan 20, 2026

View reviewed changes

DavidHuber-NOAA requested a review from aerorahul January 20, 2026 20:41

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

8b51570

DavidHuber-NOAA merged commit b86a732 into NOAA-EMC:develop Jan 21, 2026
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor Tar Archiving Logic to Python-Based Implementation #4336

Refactor Tar Archiving Logic to Python-Based Implementation #4336

AntonMFernando-NOAA commented Dec 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DavidHuber-NOAA commented Jan 14, 2026

Uh oh!

emcbot commented Jan 14, 2026

Uh oh!

DavidHuber-NOAA commented Jan 14, 2026

Uh oh!

DavidHuber-NOAA left a comment •

edited

Loading

Uh oh!

DavidHuber-NOAA commented Jan 16, 2026

Uh oh!

DavidHuber-NOAA commented Jan 20, 2026

Uh oh!

DavidHuber-NOAA commented Jan 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Refactor Tar Archiving Logic to Python-Based Implementation #4336

Refactor Tar Archiving Logic to Python-Based Implementation #4336

Conversation

AntonMFernando-NOAA commented Dec 15, 2025

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DavidHuber-NOAA commented Jan 14, 2026

Uh oh!

emcbot commented Jan 14, 2026

Uh oh!

DavidHuber-NOAA commented Jan 14, 2026

Uh oh!

DavidHuber-NOAA left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA commented Jan 16, 2026

Uh oh!

DavidHuber-NOAA commented Jan 20, 2026

Uh oh!

DavidHuber-NOAA commented Jan 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

DavidHuber-NOAA left a comment •

edited

Loading