GridDiag wrapper failure

[JohnHalleyGotway]

Moving a conversation over email with @mpm-meto into a discussion for the benefit of other users. Listed below is the original question...

I am rerunning some use cases I last ran in mid-2024 with MET11.0.0. The GridDiag compared the same variable in two different files. Like this.

GRID_DIAG_INPUT_DIR = /data/scratch/marion.mittermaier
# Note that the file names below would appear to need to be the same
GRID_DIAG_INPUT_TEMPLATE = DEMO5/nc/Fcast_{init?fmt=%Y%m%d}_{init?fmt=%02H}00_{lead?fmt=%03H}_u-dp135surf.nc,
                           DEMOctrl/nc/Fcast_{init?fmt=%Y%m%d}_{init?fmt=%02H}00_{lead?fmt=%03H}_u-dq613surf.nc

BOTH_VAR1_NAME = 'cloud_area_fraction_assuming_maximum_random_overlap'
BOTH_VAR1_LEVELS = (*,*)
BOTH_VAR1_OPTIONS = set_attr_name="DEMO5_TCA"; set_attr_level="sfc"; n_bins = 20; range = [0,1];

BOTH_VAR2_NAME = 'cloud_area_fraction_assuming_maximum_random_overlap'
BOTH_VAR2_LEVELS = (*,*)
BOTH_VAR2_OPTIONS = set_attr_name="DEMOctrl_TCA"; set_attr_level="sfc"; n_bins = 20; range = [0,1];

Unfortunately this doesn’t work. It fails with:

ERROR  :
ERROR  : get_series_entry() -> Could not find data for 'cloud_area_fraction_assuming_maximum_random_overlap'(*,*) in file list:
ERROR  : /data/scratch/marion.mittermaier/DEMO5/nc/Fcast_20210102_0000_024_u-dp135surf.nc
ERROR  :

I have belatedly concluded that the reason it worked before is that the filenames were the same but they were in different directories. The code does not appear to work when the filenames and directories are different (as above).

I don’t know how to get around this. I can’t give them the same name as the expid is important from a metadata perspective. Data volumes dictate that I can’t duplicate and rename somehow. I can only think of very hacky and very convoluted ways to proceed…. Like use ncks to extract the field from each file and save both in the same filename template in different directories… is that my only option? I guess I could also revert to just bypassing the wrapper. That should work as well.

Any thoughts?

[JohnHalleyGotway]

@mpm-meto, I see you have a question about MET's grid_diag tool or the METplus GridDiag wrapper.

I did some testing and chatted with @georgemccabe about this.

I do want to point out this important detail about MET's grid_diag tool from this section of the User's Guide.

When -data is used once, all fields are read from each input file. When used multiple times, it must match the number of fields to be processed. In this case the first field in the config data field list is read from the files designated by the first -data, the second field in the field list is read from files designated by the second -data, and so forth.

@georgemccabe confirmed that each input file passed to the GridDiag wrapper is added with it's own -data command line option. I'd assumed it was passing all input files with a single -data option, and that's what was causing the issue. But since it's not, your setup should actually work fine.

BUT @georgemccabe did see that it may be single quotes wreaking havoc. In your config settings, please try removing the single quotes from the names. So change...
FROM:

BOTH_VAR1_NAME = 'cloud_area_fraction_assuming_maximum_random_overlap'
BOTH_VAR2_NAME = 'cloud_area_fraction_assuming_maximum_random_overlap'

TO:

BOTH_VAR1_NAME = cloud_area_fraction_assuming_maximum_random_overlap
BOTH_VAR2_NAME = cloud_area_fraction_assuming_maximum_random_overlap

And rerun. Does that produce better results?

@georgemccabe and I need to brainstorm whether additional logic is needed in the GridDiag wrapper to handle the setting of the -data command line option. But hopefully your immediate issue is just with the use of quotes.