Refactor Tar Archiving Logic to Python-Based Implementation #4336

AntonMFernando-NOAA · 2025-12-15T16:00:40Z

Description

This PR refactors the tar archiving (arch_tar) workflow to move all logic, cycle variable handling, and COM path resolution from YAML/Jinja templates into Python. YAML configuration is now limited to describing what files are archived, while Python scripts are responsible for how variables are generated and resolved.
This change aligns tar archiving with the local archiving (arch_vrfy) approach and establishes a unified, Python-driven archive framework.

Resolves [Enabler] Review the archiving process and determine best paths forward #3656
Refs Archive Enhancement — Migrate Template Variables from Jinja to Python #3910

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? YES/NO (If YES, please indicate to which system(s))
- GFS
- GEFS
- SFS
- GCAFS
Is this a breaking change (a change in existing functionality)?NO
Does this change require a documentation update? /NO
Does this change require an update to any of the following submodules? NO
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

Alll available CI tests on Hera and extended tests on WCOSS2

Checklist

…do-NOAA/global-workflow into enabler/archive_tar

ush/python/pygfs/utils/archive_tar_vars.py

…do-NOAA/global-workflow into enabler/archive_tar

ush/python/pygfs/task/archive_tar_vars.py

aerorahul

looks good. Just a couple of questions.

aerorahul · 2026-01-09T16:42:54Z

ush/python/pygfs/utils/archive_tar_vars.py

+
+        # Archive timing booleans - increments (group a)
+        # Logic: (current_cycle - SDATE).days % ARCH_WARMICFREQ == 0 AND is_gdas AND ARCH_CYC == cycle_HH
+        if sdate:


When is this condition False? config_dict always contains SDATE, no? It is in the config.base

aerorahul · 2026-01-09T16:43:31Z

ush/python/pygfs/utils/archive_tar_vars.py

+
+        # Archive timing booleans - ICs (group b)
+        # Logic: (ics_offset_cycle - SDATE).days % ARCH_WARMICFREQ == 0 AND is_gdas AND (ARCH_CYC - assim_freq) % 24 == cycle_HH
+        if sdate:


@aerorahul That's correct. sdate if statement is not needed here.

DavidHuber-NOAA

Looks great! I just have a few minor suggestions/questions.

DavidHuber-NOAA · 2026-01-09T16:22:26Z

parm/archive/enkf_restarta_grp.yaml.j2

+            {% for iaufhr in IAUFHRS %}
+                {% for itile in range(6) %}
+        - "{{ COMIN_ATMOS_ANALYSIS_MEM }}/{{ head }}recentered_jedi_increment.atm.i{{ '%03d' % iaufhr }}.tile{{ itile+1 }}.nc"


Is it possible to format IAUFHRS in the Python code so you don't have to format iaufhr here? I'm not sure if that will have repercussions elsewhere...

DavidHuber-NOAA · 2026-01-09T16:39:04Z

ush/python/pygfs/utils/archive_tar_vars.py

+            # Archive timing and control
+            'ARCH_CYC', 'ARCH_WARMICFREQ', 'ARCH_FCSTICFREQ',
+            # Other
+            'NET',


NET is a basic configuration variable. It's the name of the system (GFS, GEFS, GCAFS, or SFS).

DavidHuber-NOAA · 2026-01-09T16:40:22Z

ush/python/pygfs/utils/archive_tar_vars.py

+            # Basic configuration
+            'ATARDIR', 'current_cycle', 'RUN', 'PDY', 'PSLOT',
+            # Archive control
+            'DO_ARCHCOM', 'ARCHCOM_TO', 'ROTDIR', 'PARMgfs', 'ARCDIR', 'SDATE', 'MODE',


MODE, PARMgfs, ROTDIR, and SDATE are also basic configuration.

AntonMFernando-NOAA · 2026-01-09T17:47:42Z

@aerorahul @DavidHuber-NOAA Addressed all the comments. Running all the tests on hera and wcoss2 (extended) to verify everything works.

TravisElless-NOAA

My individual testing shows things should be working correctly here.

AntonMFernando-NOAA and others added 15 commits December 9, 2025 14:53

test

a61ed7c

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

edaa085

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

31b6fc0

rename vars

6703cd0

Merge branch 'enabler/archive_tar' of https://github.com/AntonMFernan…

1dbc3dd

…do-NOAA/global-workflow into enabler/archive_tar

remove com paths from master_enkf.yaml

faf93e8

remove deepcopy

fff50be

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

461e362

cyc_vars migrations from sub-yamls

5497488

update parm/archive/master_enkf.yaml.j2

0857700

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

4776694

change yaml vars

a8c4b97

Merge branch 'enabler/archive_tar' of https://github.com/AntonMFernan…

05d4e21

…do-NOAA/global-workflow into enabler/archive_tar

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

63de9a4

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

04bcc71

AntonMFernando-NOAA self-assigned this Dec 15, 2025

aerorahul reviewed Dec 15, 2025

View reviewed changes

ush/python/pygfs/utils/archive_tar_vars.py Outdated Show resolved Hide resolved

ush/python/pygfs/utils/archive_tar_vars.py Show resolved Hide resolved

ush/python/pygfs/utils/archive_tar_vars.py Outdated Show resolved Hide resolved

AntonMFernando-NOAA and others added 3 commits December 15, 2025 13:48

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

bd6f4f1

removed _get_cyc_vars() method

19c912f

Merge branch 'enabler/archive_tar' of https://github.com/AntonMFernan…

010183b

…do-NOAA/global-workflow into enabler/archive_tar

aerorahul reviewed Dec 15, 2025

View reviewed changes

ush/python/pygfs/task/archive_tar_vars.py Outdated Show resolved Hide resolved

removed a duplicate test file

847ca33

AntonMFernando-NOAA requested a review from DavidHuber-NOAA December 15, 2025 19:16

AntonMFernando-NOAA and others added 7 commits December 15, 2025 19:33

simplification

fef813a

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

3d047a0

update archive scripts

a6e39ce

update the archive.py

6c2a48d

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

2e6dcdb

Delete verify_refactor.py

fc45122

added more cyc vars

e428f7c

AntonMFernando-NOAA and others added 7 commits December 25, 2025 18:21

typos

9fd9229

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

07bc3fa

update ush/python/pygfs/utils/archive_tar_vars.py

ac7fba8

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

da03f4c

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

b6497a2

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

33793d4

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

bde7d98

AntonMFernando-NOAA marked this pull request as ready for review December 31, 2025 15:03

Merge remote-tracking branch 'origin/develop' into enabler/archive_tar

c4139ac

AntonMFernando-NOAA force-pushed the enabler/archive_tar branch 3 times, most recently from be6a91b to c4139ac Compare January 5, 2026 19:11

AntonMFernando-NOAA and others added 6 commits January 5, 2026 19:16

git update parm/archive/gdas_restarta.yaml.j2

3907e83

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

724a946

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

6193b9c

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

071f89c

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

1a6b471

Merge branch 'NOAA-EMC:develop' into enabler/archive_tar

f60f299

aerorahul previously approved these changes Jan 9, 2026

View reviewed changes

DavidHuber-NOAA previously approved these changes Jan 9, 2026

View reviewed changes

update IAUFHRS

f1b4466

AntonMFernando-NOAA dismissed stale reviews from DavidHuber-NOAA and aerorahul via f1b4466 January 9, 2026 17:39

updated ush/python/pygfs/utils/archive_tar_vars.py

8f8d432

DavidHuber-NOAA previously approved these changes Jan 9, 2026

View reviewed changes

TravisElless-NOAA previously approved these changes Jan 9, 2026

View reviewed changes

typos

054ce01

AntonMFernando-NOAA dismissed stale reviews from TravisElless-NOAA and DavidHuber-NOAA via 054ce01 January 9, 2026 23:59

Refactor Tar Archiving Logic to Python-Based Implementation #4336

Are you sure you want to change the base?

Refactor Tar Archiving Logic to Python-Based Implementation #4336

Conversation

AntonMFernando-NOAA commented Dec 15, 2025

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

aerorahul Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

aerorahul Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

AntonMFernando-NOAA Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

AntonMFernando-NOAA Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

AntonMFernando-NOAA commented Jan 9, 2026

Uh oh!

TravisElless-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants