Add netcdf restart and history files using PIO (parallel netCDF) for dev/ufs-weather-model

[DeniseWorthen]

Is your feature request related to a problem? Please describe.

WW3 uses binary I/O. This includes the initialization file, the history files and the restart file. The mesh cap branch includes a capability for run-time output of 'gridded mean fields' in netCDF. The current netCDF history capability writes netCDF serially. Switching to PIO would allow history output in parallel, which allows for potential scalability benefits with large meshes.

The binary restart files from WW3 are difficult to debug. For large unstructured domains, the writing of restart files for UFS is also found to be too slow. Implementing PIO/netCDF restarts would resolve both issues.

Describe the solution you'd like
Implement PIO+PnetCDF capability for both run-time history files as well as restart files for the dev/ufs-weather-model branch.

[DeniseWorthen]

@mvertens @alperaltuntas I have a working branch for this capability which I've tested with both structured and un-structured meshes. I've obtained restart repro as well as invariance to MPI decomposition for both types. I've added hooks for CESM and NorESM to configure PIO through the shr code. I need to do some final testing, but I should have something that you could test and modify for your use cases soon.

[mvertens]

@DeniseWorthen - that sounds amazing! Thanks for this.

[DeniseWorthen]

@mvertens @alperaltuntas I have a clean feature branch for this work here. Please test when you get the chance.

https://github.com/DeniseWorthen/WW3/tree/feature/pio4ww3

I'm happy to have a tag-up to walk you through the changes, which are fairly substantial, primarily because I've backed out the changes we needed to make to w3wavemd and w3iorsmd for the mesh cap. Enabling netCDF restarts and history now occurs separately from any native WW3 writing or reading, so only minor modifications are needed relative to the develop branch.

In brief, there are two config parameters (use_restartnc and use_historync) which will enable the PIO+netCDF capability. These are set default true for CESM. There is also a setting which will allow you to startup from a binary WW3 restart (restart_from_binary).

The PIO is initialized in wav_pio_mod.F90. The new restart capability is in wav_restart_mod.F90. The netCDF history is now all in wav_history_mod.F90. This contains the old w3iogoncmd and the old wav_grdout routine.

[mvertens]

@DeniseWorthen - this is a huge step forwards. I'd love a walk through. @alperaltuntas - do you want to join? I can do tonight or even Friday night my time.

[alperaltuntas]

Thanks @DeniseWorthen, @mvertens. I can also join a call on Friday. I am available any time. In the meantime, I'll try testing the branch.

[DeniseWorthen]

I have one last issue to resolve before opening at PR for this work. When waves are in the slow loop, the model does not restart reproduce. All other tests pass in the UWM RTs.

The slow-loop waves fail to restart repro for either structured or unstructured meshes. I've also verified that using dev/ufs-weather-model and writing only mapsta and va to the binary restarts does restart reproduce in the slow loop. So I am not missing any needed fields in the netCDF restarts. I suspect a flag or something that is not being set correctly for the netCDF restarts.