PcpCombine won't open NetCDF files that GridStat opens

[michellefrazer]

I am using MET 11.0.3 and METplus 5.0.2.

I am trying to use PcpCombine on some NetCDF files that I had no issues previously running GridStat (and other processes) on.

For GridStat, I use "FCST_GRID_STAT_FILE_TYPE = NETCDF_NCCF". But for PcpCombine, it seems there is no direct equivalent, but I have tried "FCST_PCP_COMBINE_INPUT_DATATYPE = NETCDF" and "FCST_PCP_COMBINE_INPUT_DATATYPE = NETCDF_NCCF". However, I've noticed the "INPUT_DATATYPE" config option doesn't change the actual MET command that is run. I have also tried "FCST_PCP_COMBINE_INPUT_OPTIONS = file_type = NETCDF_NCCF;", so running a command like:

/discover/swdev/sdrabenh/my_libs/libs-v07-mil_gnu-13.2.0/bin/pcp_combine -v 4 -sum 20250501_000000 01 20250501_060000 06 -pcpdir /discover/nobackup/projects/gmao/osse2/HWT/CONUS02KM/Feature-c2160_L137/forecasts/CYCLED_REPLAY_P10800_C21600_T21600_20250501_00z -pcprx GEOS\.hwt_01hr_slv_LCC\.20250501_0600z\.nc4 -field 'name="APCP_1H"; level="(0,*,*)"; file_type = NETCDF_NCCF;' -name "APCP_6H" testing/output/GEOS/PCPCombine/20250501_06

I get the following error:

DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_None".
WARNING: search_pcp_dir() -> can't open data file "/discover/nobackup/projects/gmao/osse2/HWT/CONUS02KM/Feature-c2160_L137/forecasts/CYCLED_REPLAY_P10800_C21600_T21600_20250501_00z/GEOS.hwt_01hr_slv_LCC.20250501_0600z.nc4"
ERROR  :
ERROR  : sum_data_files() -> cannot find a file with a valid time of 20250501_060000 and accumulation time of 010000 matching the regular expression "GEOS.hwt_01hr_slv_LCC.20250501_0600z.nc4"
ERROR  :

Any suggestions?

Thanks!

[j-opatz]

Hi Michelle,

And thank you for your question. It looks like you were on the right track: FCST_PCP_COMBINE_INPUT_OPTIONS = file_type = NETCDF_NCCF; is the correct variable setting to use in PCPCombine to alert MET that the input file is a netCDF file following CF-conventions. Without getting into the finer details on it, FCST_PCP_COMBINE_INPUT_DATATYPE is an older setting that is meant to support GEMPAK data conversion, and does not actually affect the file_type setting in MET.

As for why you are running into an error, I cannot be certain. The command you provided should work as intended, as long as the file has a valid time matching the one you pass on command line (20250501_060000). I'm going to request that you send along a config file, along with some test data, so that we can properly assess what's giving you these errors. Information on how to send us data can be found at this link.

A final note is that the version of MET and METplus you are running, 11.0.3 and 5.0.2 respectively, are a bit behind the latest releases (12.2.0 and 6.2.0). I would strongly suggest that you try and upgrade to the latest releases, as many of the errors and issues that users tend to find in older versions were actually fixed in newer releases.

[michellefrazer]

Thanks! I've attached two config files, the one I'm trying to use for PCPcombine (resulting in the file open error) and a dummy GridStat one that successfully opens the same forecast files. I've attached the netcdf file as well, all with extension changes.
test_GridStat.txt
test_PCPCombine.txt
GEOS.hwt_01hr_slv_LCC.20250501_1200z.txt

[j-opatz]

Hi Michelle,

Thank you for providing the test data! I was able to test the file you sent over and I think the problem boils down to projection issues.

My first test is always to see if the input data file projects correctly using the Plot-Data-Plane tool. If MET cannot recognize the file, or has difficulty projecting the results correctly in Plot-Data-Plane, any MET tool usage cannot be trusted (as interpolation/distance methods may not be correctly done).

I did need to add the file_type=NETCDF_NCCF flag to get the file through Plot-Data-Plane which isn't necessarily an issue, but the results were concerning. Take a look at the image I got:

You can see in the lower left of the plot there are some black squiggles? That's actually the entire globe's projection. This means that even though the data field shows up front and center in plot, the data itself is not being projected correctly in MET. I'm very surprised that you were able to previously run this type of input file through MET tools; any statistics you get would be suspect as MET is not properly projecting the data fields in the file. We have had users successfully use MET with data input that does not properly project onto a map, but we cannot offer support or scientific integrity to the results because METplus is not being used as designed.

Take a look at the User's Guide section on supported grid projections. If the data file's projection system is not on this list, you will either need to reproject it to a supported grid outside of MET (the Regrid-Data-Plane tool only supports regridding between grids that MET supports), or utilize the UGRID functionality to read the file in as a unstructured grid. More information on UGRID functionality is here in the User's Guide.

[michellefrazer]

Thanks, I should have been clearer in what I was and was not able to do. Previously, I have used GridStat to run stats on our standard "GEOS-FP" model and some AI foundation models for comparison. Those are all on standard Lat/Lon grids. (As a side note, we were never able to adopt MET for our workflow because of the area-weighted regridding bug: dtcenter/MET#3178).

The data I sent you before is from our hi-res 2km model and was on Lambert grid as follows: "Lambert Conformal Nx: 1799 Ny: 1059 Lat_LL: 21.138 Lon_LL: 122.720 Lon_orient: 97.500 Alpha: 4203.494 Cone: 0.623 Bx: 899.1693 By: 3198.9984" (that come from the log of my coworker using MET to regrid some observation data to the same grid).

The 2km model is the one with the the APCP data we want to sent through PcpCombine before GridStat. However, PcpCombine can't even open the file as you saw. Our current workaround is to convert the files to GRIB before running through PcpCombine. This works great and we get reasonable results, but it isn't efficient.

I had also thought this might be a projection issue so I ran two more tests. First, I tried to open the Lambert grid 2km model file in GridStat and it worked (at least it opened -- didn't get to results as I just used the dummy config I shared with you). Then, I tried using PcpCombine on the standard Lat/Lon GEOS-FP (which does produce good GridStat results, but doesn't have APCP data) and it would open. It produced the same error I shared in my original post.

So in case there is a legit bug in PcpCombine keeping it from opening files that GridStat reads corrects, I have attached a sample "GEOS-FP" netcdf file for you to test. I should have sent you this as well last time!
GEOS.fp.fcst.inst3_3d_asm_Np.20241201_00+20241201_0000.txt

Thank you!

[j-opatz]

Hello Michelle,

And thank you for the additional details and test file.

I've tested the newest file and confirmed that Plot-Data-Plane projects this file correctly, although it does require the file_type setting to properly read in the data, i.e. this is not CF-compliant. Here is the image the tool creates; note that the data is properly projected on the globe:

As I understand the issue, the first GEOS-HWT file (2km resolution) you sent cannot be read by PCPCombine as it is. Currently, you have a workaround to convert the file to GRIB format, but that is not desirable. Additionally, the second file you provided CAN go through GridStat, but has no need for PCPCombine as it does not have precipitation data. The end goal is to get the GEOS-HWT file read in by PCPCombine. Is this a correct summary of how things stand?

Knowing that the first GEOS-HWT file is a Lambert Conformal projection, while the second GEOS-FP file is a standard Lat/Lon grid, has solidified my answer that this is a projection/file structure issue. The "bug" you're seeing with PCPCombine is MET failing to understand how to read the projection of the input file since we are telling MET that the input file is CF-compliant, when it really isn't.

I ran the GEOS-HWT file through a CF convention check website and found that it failed in a few areas: most notably, it does not have any grid_mapping attributes, so MET would not know the file is in a Lambert Conformal projection. I confirmed this when I checked the log output for a Plot-Data-Plane on the GEOS-HWT file:

DEBUG 4: Grid information:
DEBUG 4:    Projection: Lat/Lon
DEBUG 4:    Nx: 1799
DEBUG 4:    Ny: 1059
DEBUG 4:    lat_ll: 1.000
DEBUG 4:    lon_ll: -1.000
DEBUG 4:    delta_lat: 1.000
DEBUG 4:    delta_lon: 1.000
DEBUG 4:    wrapLon: true
DEBUG 4:

Note how it counts the projection as Lat/Lon, rather than Lambert Conformal. This is MET defaulting to Lat/Lon, as it has no grid_mapping information in the netCDF file. So if you are currently able to read this style of file into any MET tool, it's more or less a fluke; the output is questionable as MET cannot properly project the data onto a gridded map.

This also explains why the GEOS-FP file succeeds in Plot-Data-Plane (and any other MET tool): the data is in Lat/Lon projection, which is MET's default, so MET guessing ends up being correct and verification can continue.

I ran a CF-compliant file through Plot-Data-Plane, which also uses a Lambert-Conformal projection, and the log file output for Grid information says it all:

DEBUG 4: Grid information:
DEBUG 4:    Projection: Lambert Conformal
DEBUG 4:    Nx: 451
DEBUG 4:    Ny: 337
DEBUG 4:    Lat_LL: 16.281
DEBUG 4:    Lon_LL: 126.138
DEBUG 4:    Lon_orient: 95.000
DEBUG 4:    Alpha: 1220.460
DEBUG 4:    Cone: 0.423
DEBUG 4:    Bx: 246.0052
DEBUG 4:    By: 1052.1944
DEBUG 4:

I'll also copy the ncdump results of the input file, so you can see the difference between this compliant file and the GEOS-HWT file:

netcdf gtg_obs_forecast.20130404.i12.f06 {
dimensions:
	y = 337 ;
	x = 451 ;
	z = 51 ;
	name_len = 32 ;
	time = 1 ;
variables:
	int grid_mapping ;
		grid_mapping:grid_mapping_name = "lambert_conformal_conic" ;
		grid_mapping:standard_parallel = 25. ;
		grid_mapping:longitude_of_central_meridian = -95. ;
		grid_mapping:latitude_of_projection_origin = 25. ;
		grid_mapping:false_easting = 0 ;
		grid_mapping:false_northing = 0 ;
		grid_mapping:GRIB_earth_shape = "spherical" ;
		grid_mapping:GRIB_earth_shape_code = 0 ;
	double time(time) ;
		time:long_name = "forecast time" ;
		time:units = "seconds since 1970-1-1" ;
		time:axis = "T" ;
	double forecast_reference_time ;
		forecast_reference_time:long_name = "reference time" ;
		forecast_reference_time:units = "seconds since 1970-1-1" ;
	int forecast_period ;
		forecast_period:units = "seconds" ;
	double x(x) ;
		x:standard_name = "projection_x_coordinate" ;
		x:units = "m" ;
		x:grid_spacing = "13545.087 m" ;
		x:axis = "X" ;
	double y(y) ;
		y:standard_name = "projection_y_coordinate" ;
		y:units = "m" ;
		y:grid_spacing = "13545.087 m" ;
		y:axis = "Y" ;
	double lat(y, x) ;
		lat:units = "degrees_north" ;
		lat:long_name = "latitude coordinate" ;
		lat:standard_name = "latitude" ;
		lat:_CoordinateAxisType = "Lat" ;
	double lon(y, x) ;
		lon:units = "degrees_east" ;
		lon:long_name = "longitude coordinate" ;
		lon:standard_name = "longitude" ;
		lon:_CoordinateAxisType = "Lon" ;
	double z(z) ;
		z:standard_name = "altitude" ;
		z:long_name = "flight_level" ;
		z:units = "feet" ;
		z:positive = "up" ;
		z:axis = "Z" ;
	int field_id ;
		field_id:long_name = "field indentification number" ;
	char field_name(name_len) ;
		field_name:long_name = "field name" ;
	float edr(time, z, y, x) ;
		edr:units = "m^(2/3) s^-1" ;
		edr:long_name = "Eddy dissipation rate" ;
		edr:coordinates = "lat lon" ;
		edr:_FillValue = -9999.f ;
		edr:grid_mapping = "grid_mapping" ;

// global attributes:
		:Conventions = "CF-1.4" ;
		:Originating_center = "NCAR" ;
		:Product_Status = "Experimental Products" ;
		:Product_Type = "Forecast/Uninitialized Analysis" ;
		:DataType = "GRID" ;
		:DatasetLocation = "" ;
}

So until your GEOS file(s) are in CF-compliant format, you will continue to need to utilize some post-processing outside of MET to attach grid information, such as Python Embedding. Otherwise, MET cannot read the files except by accident.

[michellefrazer]

Thanks, I think what I'm trying to say is that neither file can be opened by PCPCombine ("WARNING: search_pcp_dir() -> can't open data file"). I'm making a distinction between the file being opened in the first place, and whether or not MET can find the right variable, lead time, etc. Those two issues produces very different errors in the log. Both files can be opened by GridStat. Neither can be opened by PCPCombine. Something is different about how PCPCombine handles NetCDF files compared to GridStat.

[j-opatz]

Thank you for your patience Michelle,

I wanted to ensure we got to the bottom of the problem and I think we have.

I ran both files you provided (GEOS hwt and fp) through PCPCombine to again confirm that both files end up with 2 warnings: one for not being able to open the file, and one for not finding the correct data. It's that first warning, not being able to open the file, that we wanted to understand a bit more about, due to the same files being opened by MET via Grid-Stat.

After speaking with our lead MET engineer, it seems that this comes down to a bit of coding in PCP-Combine, along with an understanding of the tool differences. It turns out that the sum argument is unique in PCP-Combine in that it does not open a specific file name. Grid-Stat opens a specific file name, as do the other PCP-Combine options (add, subtract, and derive). But the PCP-Combine sum option is different. It searches the directory checking whether or not each file contains the data with the timing info it needs. So a warning is produced both for the timing issue, as well as not being CF-compliant.

But to really understand how the warning arises of PCP-Combine not being able to open your file, take a look at this line of code which attempts to open the file for reading. The search_pcp_dir() function loops over the files in the input directory.

Line 788 attempts to open each file to be considered. Note that it does NOT pass in a user-provided "file type". So even when supplying file_type = NETCDF_NCCF;, that is not passed to that function. That functionality works fine for any GRIB1, GRIB2, or even NetCDF file that MET knows how to identify as being CF-compliant or following the MET format.

But your files are not identified as such, which is the reason why that warning message is being printed:

search_pcp_dir() -> can't open data file ...

We are looking into adding capability that would allow PCP-Combine to read in file_type settings, which would remove the warning you are concerned about. That should be in a MET 12.2 bugfix that is being worked on at the moment, but I can report back when the code is actually in place.

I again want to emphasize that while this would clear up one of the warnings, both files are not CF-compliant and will raise other issues and warnings during the verification process.

[JohnHalleyGotway]

FYI, this PCP-Combine issue about using the file_type when searching for matching files for the sum command (dtcenter/MET#1533) was fixed today, merged into the main_v12.2 branch (dtcenter/MET#3354), and included in the MET-12.2.1 bugfix release.

@michellefrazer, if you have the opportunity to test out that bugfix release and check whether PCP-Combine behaves as you'd expect it to, we'd love to hear feedback! Thanks for reporting this issue and please keep the feedback coming.

[michellefrazer]

Thanks all! I'll talk to my co-worker about updating our MET build and testing. But due to the issues opening our non-CF-compliant Lambert grid files, we'll probably stick with converting to Grib for now. ;) Thanks again!