[5.5.2]: Not all ensemble members can not be readed as ens coordinate in GRIB 2 file

[Yaqiang]

Versions impacted by the bug

v5.x

What went wrong?

I have a grid2 data file which contains 4 ensemble members: NCEP, JMA, Beijing and ECMWF. Using ToolUI 5.5.2 only two ensemeble members are extracted with ens coordinate length of 2. The example data file can be downloaded from here: http://www.meteothink.org/downloads/data/201809_12.grib.

The data file information extracted from NCL:

Variable: f
Type: file
filename: 201809_12
path: 201809_12.grib
file global attributes:
dimensions:
initial_time0_hours = 30
forecast_time0 = 5
lat_0 = 121
lon_0 = 161
variables:
float 10u_P1_L103_GLL0 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
center : US National Weather Service - NCEP (WMC)
production_status : TIGGE Operational products
long_name : 10 meter u velocity
units : m/s
_FillValue : 1e+20
grid_type : Latitude/longitude
parameter_discipline_and_category : Meteorological products, Momentum
parameter_template_discipline_category_number : ( 1, 0, 2, 2 )
level_type : Specified height level above ground (m)
level : 10

  float 10v_P1_L103_GLL0 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
     center :       US National Weather Service - NCEP (WMC)
     production_status :    TIGGE Operational products
     long_name :    10 meter v velocity
     units :        m/s
     _FillValue :   1e+20
     grid_type :    Latitude/longitude
     parameter_discipline_and_category :    Meteorological products, Momentum
     parameter_template_discipline_category_number :        ( 1, 0, 2, 3 )
     level_type :   Specified height level above ground (m)
     level :        10

  float 10u_P1_L103_GLL0_1 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
     center :       Japanese Meteorological Agency - Tokyo (RSMC)
     production_status :    TIGGE Operational products
     long_name :    10 meter u velocity
     units :        m/s
     _FillValue :   1e+20
     grid_type :    Latitude/longitude
     parameter_discipline_and_category :    Meteorological products, Momentum
     parameter_template_discipline_category_number :        ( 1, 0, 2, 2 )
     level_type :   Specified height level above ground (m)
     level :        10

  float 10v_P1_L103_GLL0_1 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
     center :       Japanese Meteorological Agency - Tokyo (RSMC)
     production_status :    TIGGE Operational products
     long_name :    10 meter v velocity
     units :        m/s
     _FillValue :   1e+20
     grid_type :    Latitude/longitude
     parameter_discipline_and_category :    Meteorological products, Momentum
     parameter_template_discipline_category_number :        ( 1, 0, 2, 3 )
     level_type :   Specified height level above ground (m)
     level :        10

  float 10u_P1_L103_GLL0_2 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
     center :       Beijing (RSMC)
     production_status :    TIGGE Operational products
     long_name :    10 meter u velocity
     units :        m/s
     _FillValue :   1e+20
     grid_type :    Latitude/longitude
     parameter_discipline_and_category :    Meteorological products, Momentum
     parameter_template_discipline_category_number :        ( 1, 0, 2, 2 )
     level_type :   Specified height level above ground (m)
     level :        10

  float 10v_P1_L103_GLL0_2 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
     center :       Beijing (RSMC)
     production_status :    TIGGE Operational products
     long_name :    10 meter v velocity
     units :        m/s
     _FillValue :   1e+20
     grid_type :    Latitude/longitude
     parameter_discipline_and_category :    Meteorological products, Momentum
     parameter_template_discipline_category_number :        ( 1, 0, 2, 3 )
     level_type :   Specified height level above ground (m)
     level :        10

  float 10u_P1_L103_GLL0_3 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
     center :       European Center for Medium-Range Weather Forecasts
     production_status :    TIGGE Operational products
     long_name :    10 meter u velocity
     units :        m/s
     _FillValue :   1e+20
     grid_type :    Latitude/longitude
     parameter_discipline_and_category :    Meteorological products, Momentum
     parameter_template_discipline_category_number :        ( 1, 0, 2, 2 )
     level_type :   Specified height level above ground (m)
     level :        10

  float 10v_P1_L103_GLL0_3 ( initial_time0_hours, forecast_time0, lat_0, lon_0 )
     center :       European Center for Medium-Range Weather Forecasts
     production_status :    TIGGE Operational products
     long_name :    10 meter v velocity
     units :        m/s
     _FillValue :   1e+20
     grid_type :    Latitude/longitude
     parameter_discipline_and_category :    Meteorological products, Momentum
     parameter_template_discipline_category_number :        ( 1, 0, 2, 3 )
     level_type :   Specified height level above ground (m)
     level :        10

  double initial_time0_hours ( initial_time0_hours )
     long_name :    initial time
     units :        hours since 1800-01-01 00:00

  double initial_time0_encoded ( initial_time0_hours )
     long_name :    initial time encoded as double
     units :        yyyymmddhh.hh_frac

  float lat_0 ( lat_0 )
     long_name :    latitude
     grid_type :    Latitude/Longitude
     units :        degrees_north
     Dj :   0.5
     Di :   0.5
     Lo2 :  140
     La2 :   0
     Lo1 :  60
     La1 :  60

  float lon_0 ( lon_0 )
     long_name :    longitude
     grid_type :    Latitude/Longitude
     units :        degrees_east
     Dj :   0.5
     Di :   0.5
     Lo2 :  140
     La2 :   0
     Lo1 :  60
     La1 :  60

  integer forecast_time0 ( forecast_time0 )
     long_name :    Forecast offset from initial time
     units :        hours

  string initial_time0 ( initial_time0_hours )
     long_name :    Initial time of first record
     units :        mm/dd/yyyy (hh:mm)

The data file information extracted from NCDump of ToolUI:

netcdf D:/Temp/grib/201809_12.grib {
dimensions:
lon = 161;
lat = 121;
reftime = 30;
timeOffset = 5;
height_above_ground = 1;
ens = 2;
variables:
int LatLon_Projection;
:grid_mapping_name = "latitude_longitude";
:earth_radius = 6371229.0; // double

float lat(lat=121);
  :units = "degrees_north";

float lon(lon=161);
  :units = "degrees_east";

double reftime(reftime=30);
  :units = "Hour since 2018-09-01T12:00:00Z";
  :standard_name = "forecast_reference_time";
  :long_name = "GRIB reference time";
  :calendar = "proleptic_gregorian";

double timeOffset(timeOffset=5);
  :_CoordinateAxisType = "TimeOffset";
  :units = "Hour";
  :standard_name = "forecast_period";
  :long_name = "time offset from runtime";
  :udunits = "Hour since 2018-09-01T12:00:00Z";

double time(reftime=30, timeOffset=5);
  :units = "Hour since 2018-09-01T12:00:00Z";
  :standard_name = "time";
  :long_name = "GRIB forecast or observation time";
  :calendar = "proleptic_gregorian";
  :_CoordinateAxisType = "Time";

float height_above_ground(height_above_ground=1);
  :units = "m";
  :long_name = "Specified height level above ground";
  :positive = "up";
  :Grib_level_type = 103; // int
  :datum = "ground";

int ens(ens=2);
  :_CoordinateAxisType = "Ensemble";

float u-component_of_wind_height_above_ground_ens(reftime=30, timeOffset=5, ens=2, height_above_ground=1, lat=121, lon=161);
  :long_name = "u-component of wind @ Specified height level above ground";
  :units = "m/s";
  :description = "u-component of wind";
  :missing_value = NaNf; // float
  :grid_mapping = "LatLon_Projection";
  :coordinates = "reftime time timeOffset ens height_above_ground lat lon ";
  :Grib_Variable_Id = "VAR_0-2-2_L103";
  :Grib2_Parameter = 0, 2, 2; // int
  :Grib2_Parameter_Discipline = "Meteorological products";
  :Grib2_Parameter_Category = "Momentum";
  :Grib2_Parameter_Name = "u-component of wind";
  :Grib2_Level_Type = 103; // int
  :Grib2_Level_Desc = "Specified height level above ground";
  :Grib2_Generating_Process_Type = "Ensemble forecast";
  :Grib2_Statistical_Process_Type = "UnknownStatType--1";

float v-component_of_wind_height_above_ground_ens(reftime=30, timeOffset=5, ens=2, height_above_ground=1, lat=121, lon=161);
  :long_name = "v-component of wind @ Specified height level above ground";
  :units = "m/s";
  :description = "v-component of wind";
  :missing_value = NaNf; // float
  :grid_mapping = "LatLon_Projection";
  :coordinates = "reftime time timeOffset ens height_above_ground lat lon ";
  :Grib_Variable_Id = "VAR_0-2-3_L103";
  :Grib2_Parameter = 0, 2, 3; // int
  :Grib2_Parameter_Discipline = "Meteorological products";
  :Grib2_Parameter_Category = "Momentum";
  :Grib2_Parameter_Name = "v-component of wind";
  :Grib2_Level_Type = 103; // int
  :Grib2_Level_Desc = "Specified height level above ground";
  :Grib2_Generating_Process_Type = "Ensemble forecast";
  :Grib2_Statistical_Process_Type = "UnknownStatType--1";

// global attributes:
:Originating_or_generating_Center = "Beijing (RSMC)";
:Originating_or_generating_Subcenter = "0";
:GRIB_table_version = "4,0";
:Type_of_generating_process = "Ensemble forecast";
:file_format = "GRIB-2";
:Conventions = "CF-1.6";
:history = "Read using CDM IOSP GribCollection v3";
:featureType = "GRID";
}

Relevant stack trace

No response

Relevant log messages

No response

If you have an example file that you can share, please attach it to this issue.

If so, may we include it in our test datasets to help ensure the bug does not return once fixed?
Note: the test datasets are publicly accessible without restriction.

Yes

Code of Conduct

• I agree to follow the UCAR/Unidata Code of Conduct

[JohnLCaron]

Hi Yaqiang:

If you look at that file in ToolsUI Isop/Grib2/Grib2Collection (see attached screenshot), you see that there are indeed 4 records for each forecast time. However, they all have pertN (perturbation number) = 0. Netcdf-Java creates a different ensemble coordinate for each unique combination of PerturbationType and PerturbationNumber. In your file, there are only two: 0,0 and 0,1.

see CoordinateEns.java:

return new EnsCoordValue(pdse.getPerturbationType(), pdse.getPerturbationNumber());

A solution if you can recode the file is to put a different PerturbationNumber (0, 1, 2, 3) for each of the 4 models, for all of the records.

John

[Yaqiang]

Hi John,

Thanks for your information! I will report it to the data provider. But I can not recode the file, so I want to know is it possible to read all 4 ensemble data using NetCDF java library for present data file?

Yaqiang

[JohnLCaron]

Hi Yaqiang:

No its not possible with the current version. One could modify the code to include the getNumberEnsembleForecasts() field in the definition of the EnsCoordValue, eg:

new EnsCoordValue(pdse.getPerturbationType(), pdse.getPerturbationNumber(), pdse.getNumberEnsembleForecasts());

However, its not clear if that would break other ensemble models. Unfortunately, the GRIB specification is all but silent on the semantics of the fields it defines. Its left to individual providers and libraries to guess what each other intended.

You could fork the code and build a version of it that would work as you ask. Or petition Unidata and convince them to do that. Or use a special tool to modify the getPerturbationNumber field as I indicated.

Sorry I dont have a better answer.

John

[Yaqiang]

Hi John,

Thanks for your suggestion! ensNumber was added in EnsCoordValue to extract all ensemble members in my case. EnsCoordValue.java, CoordinateEns.java and GribCollectionBuilderFromIndex.java files were revised (https://github.com/Unidata/netcdf-java/compare/maint-5.x...Yaqiang:maint-5.x?expand=1). Now all 4 ensemble coordinate values can be extracted from the data file.

But, you can see my RP that only "public Object extract(Grib2Record gr) " function in CoordinateEns.java can create EnsCoordValue object using the 3th argument with ensemble number. I can't find the value in other EnsCoordValue object creation functions, so I have to replace ensemble number with other values. For example, in readCoord function of GribCollectionBuilderFromIndex.java "i + 1" was used for forged ensemble number.

[Yaqiang]

This issue has been solved by PR #1192.