Parallel reading fcst files in EnsembleStat with Python embeddings

[pratiman-91]

I am using METplus for verification for ensemble. I am using python embeddings for reading fcst files. Currently the workflow reads each file one by one. Is it possible to read the fcst files in parallel or just all files once.

[j-opatz]

Hello,

And thank you for your question. What MET tool are you using for your verification purposes? And are you accessing the MET tool directly, or are you utilizing the Python wrapper versions? I will assume you are accessing Ensemble-Stat directly for the sake of this discussion, but we can also discuss other tools/methods if you would like.

Because you are utilizing Python Embedding, there are a few options to speed up the process, or pass the ensemble member files to MET in a more compact way. I'll discuss both of these options and you can pick one (or both) if they can be used in your current setup.

The first is the speed option. As of MET version 12.1, many of the tools in MET can be sped up using the OMP_NUM_THREADS setting. You can read more about this setting in the User's Guide here, but the short summary is that by utilizing additional threads in your computing environment, any parts of the verification code that can be parallelized, will be parallelized. This should result in a measurable positive impact on your runtime.

A solution could also be to utilize the ens_member_ids setting in Ensemble-Stat (or Gen-Ens-Prod). This was designed for situations where all of the ensemble members are in the same input file. So rather than having the tool read in multiple files (aka the ensemble members) one at a time, it would read in all the members at once and process the individual members using the variable MET_ENS_MEMBER_ID, which can be iterated over similarly to the * character used for latitude and longitude. More discussion on how this setting works is in the User's Guide here (you'll have to scroll down a bit to see the discussion on ens_member_ids). Note that because you are using Python Embedding to read in these files, you would need to reconfigure your Python script so that it reads all of the ensemble member files, then passes MET in memory an array that has 3 iterable dimensions: latitude, longitude, and ensemble members. In that example, your field in the Ensemble-Stat configuration file would look something like

fcst = {
  file_type = PYTHON_NUMPY;
  field = [
    {
      name  = "python_script.py MET_PYTHON_INPUT_ARG";
      level = "(MET_ENS_MEMBER_ID,*,*)";
    }
  ];
}

The MET_PYTHON_INPUT_ARG would correlate with the file names you pass Ensemble-Stat on the command line at runtime. You will also need a ens_member_ids array that tells MET how many ensemble members to expect. Make sure to start at 0 and only list values until the total value count is the same as the number of ensemble members you passed to Ensemble-Stat. For example, if you had 6 ensemble members, it would look like

ens_member_ids = [0,1,2,3,4,5];

Please let me know if you have any additional questions on this!