Replace Links with cpfs/cpreq in Workflow Scripts

[AntonMFernando-NOAA]

Description

Some jobs in the current global-workflow create links from their working directories to files in the COM directory, allowing output to be written or accessed directly in COM.

• Use cpreq when files are copied from COM directory.

• Use cpfs when files are copied to COM directory

  Refs [NCO Bug] Replace links with copy/move/rsync in workflow scripts #712

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? YES/NO (If YES, please indicate to which system(s))
  • GFS
  • GEFS
  • SFS
  • GCAFS
• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
  • EMC verif-global
  • GDAS
  • GFS-utils
  • GSI
  • GSI-monitor
  • GSI-utils
  • UFS-utils
  • UFS-weather-model
  • wxflow

How has this been tested?

• CI tests on hera and extended tests on wcoss2

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[DavidHuber-NOAA]

Looks good. There's just one member loop that should stay as links and I spotted a COMIN that should be a COMOUT.

[DavidHuber-NOAA]

Since these are member forecasts, they can stay as links for the post job.

[DavidHuber-NOAA]

This should be updated to COMOUT.

[DavidHuber-NOAA]

Alright, sorry for the whiplash. This will take some more thought. Please revert these cpreq calls to NLN and add a TODO. I will create a follow-up issue.

[aerorahul]

I don't think this is the intended use of cpreq with -R to copy a directory.
A required directory may exist, but may not contain all the required files. cpreq operates on a file and if it does not exist or is corrupted, reports an error.

[DavidHuber-NOAA]

@aerorahul I'm not sure if this is the intended purpose, but the cpreq script does take $* as the argument to cp (cpreq#L3):

cp $*

Thus, the -R does get passed as an argument to cp and if anything fails in the copy, it will call err_exit. That said, the standards only have one example with a single file operation:

   cpreq $COMIN/inputfile inputfile       # copy essential input files into working directory

All of that said, I think these linking/copying operations need to be reworked anyway and I'll open a follow-up issue for that.

[DavidHuber-NOAA]

Oh, I misread your comment. You're correct about the potentially-missing files. Still, I'll open another issue.

[aerorahul]

Yes. cpreq is essentially this: cp $*; if [[ $? -ne 0 ]]; then echo "we have a problem"; exit 1; fi

I think that's quite silly. Instead, it should work on file-by-file basis, because a directory may exist with none of the required files and yet cpreq will succeed.

Instead, we should propose to NCO that cpreq should be more thorough.

[DavidHuber-NOAA]

This should be reverted to linking.