Staging IC's

[mkbenneh]

Currently been hit with an error I assume has to do with staging the data for IC's.
I am currently using the data path provided here (https://global-workflow.readthedocs.io/en/latest/init.html) for Orion to make my test run. It turns out to fail. I have tried solving this by looking through the scripts to see how it is reading the IC files but still being hit with the same error.

This is the path to experiment, "/work2/noaa/uswrp-hu/mbenneh/global-workflow/expdir/dust".

I will be happy to get your assistance for this.

Regards!

[KateFriedman-NOAA]

@mkbenneh Can you open your files for viewing? I get "permission denied" trying to cd to your experiment directory. Check permissions all the way up to the top $USER folder to make sure the 'r' is set for users to read your files. MSU has restricted permissions by default. Thanks!

[mkbenneh]

@KateFriedman-NOAA

Could you please check now.

[KateFriedman-NOAA]

Sorry, was on leave until today. I am still getting permission denied.

[mkbenneh]

@KateFriedman-NOAA, I made it recursive for all my files in the directory this time. I believe it should work this time.

[KateFriedman-NOAA]

Thanks, I can read your experiment files now. The initial experiment job is failing because it can't read the main configuration file (config.base). From the log:

++ jjob_header.sh[104]export EXPDIR=../../expdir/dust
++ jjob_header.sh[104]EXPDIR=../../expdir/dust
++ jjob_header.sh[105]for config in "${configs[@]:-''}"
++ jjob_header.sh[106]source ../../expdir/dust/config.base
/work2/noaa/uswrp-hu/mbenneh/global-workflow/ush/jjob_header.sh: line 106: ../../expdir/dust/config.base: No such file or directory
++ jjob_header.sh[107]export err=1
++ jjob_header.sh[107]err=1
++ jjob_header.sh[108]set +x

    -------------------------------------------------------------
    -- FATAL ERROR: FATAL ERROR [/work2/noaa/uswrp-hu/mbenneh/global-workflow/ush/jjob_header.sh]: Unable to load config config.base RETURN CODE 1
    -- ABNORMAL EXIT at Fri Apr 11 01:44:12 PM CDT 2025 on orion-dtn-2
    -------------------------------------------------------------

I see EXPDIR=../../expdir/dust, which is set in config.base:

export EXPDIR="../../expdir/${PSLOT}"

It is best to set EXPDIR with the absolute path or at least with a variable that is defined as an absolute path. Otherwise, when the job runs the EXPDIR isn't correct because the job isn't running within the global-workflow clone so the job doesn't see the config.

Please change EXPDIR in your config.base to the full path to your EXPDIR:

export EXPDIR="/work2/noaa/uswrp-hu/mbenneh/global-workflow/expdir/${PSLOT}"

After your EXPDIR in config.base is updated, run the xml generator script to also update the EXPDIR references within your experiment xml:

./setup_xml.py PATH_TO_EXPDIR

If your clone is from before the recent file moment in the repo then that script is under the /workflow folder, otherwise you'll find it in the dev/workflow folder.

Then retry your experiment by either rewinding the failed job or, since you're at the very beginning of your experiment, just delete the *.db files in your EXPDIR and run rocotorun to start over. Let me know how it goes. Thanks!

Out of curiousity, what was/were your setup commands? In particular, what did you provide for --expdir if you used the setup scripts? I am guessing you provided --expdir with a path that contained the ../../?

[mkbenneh]

@KateFriedman-NOAA

Beyond the ../ci/cases/pr/C48_ATM.yaml case, are there any others with available ICS, or could you lead me to an archive where I can conduct my own test with different configurations?

[DavidHuber-NOAA]

@mkbenneh All of the YAML cases have initial conditions staged on all tier 1 machines. Seeing that you are working on MSU, you should be good to launch any of them.

[mkbenneh]

@DavidHuber-NOAA,
I used this
pslot=YOUR_TEST_NAME HPC_ACCOUNT=YOUR_COMPUTE_ACCOUNT RUNTESTS=YOUR_RUN_SPACE ./create_experiment.py --yaml ../ci/cases/pr/C48_ATM.yaml
per Kate's recommendation to run the C48_ATM.yaml case. I tested for the other cases but ran into forecast errors.

[DavidHuber-NOAA]

@mkbenneh Can you share the location of or upload your log?

[mkbenneh]

@DavidHuber-NOAA ,
My log file path here:
/work2/noaa/uswrp-hu/mbenneh/global-workflow/COMROOT/tests/logs/2021032312

[DavidHuber-NOAA]

@mkbenneh I don't see an obvious problem, but it's not an issue with the ICs. That would have caused an error much earlier.

That said, It looks like you are using a fairly old hash of the global workflow. Could you try cloning, building, and linking a fresh copy and then try again?

[mkbenneh]

@DavidHuber-NOAA,
I recloned and tried the test for cases in /work2/noaa/uswrp-hu/mbenneh/global-workflow/dev/ci/cases/pr but a couple of them fail, namely; C48_S2SWA_gefs, C48_S2SW_extended, C96_atm3DVar. The path to the experiment is here, /work2/noaa/uswrp-hu/mbenneh/global-workflow/EXPDIR and the log files can be found here; /work2/noaa/uswrp-hu/mbenneh/global-workflow/COMROOT.

[mkbenneh]

@KateFriedman-NOAA and @DavidHuber-NOAA,

I see where my issues are, and I have resolved them.

1. Initially, my input data for my IC's were directly pointed at the data path provided on the website, but I believe it had to do with permissions, so I copied it to my space and tackled user permissions, and it's all good now.

2. Had to do with the resolution, which I resolved.

I appreciate your support!