Enable to read and process the IMS data in netcdf format #2023
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds netCDF format support to the IMS (Ice Mapping System) preprocessing code, which previously only supported ASCII format input files. This enhancement is required for reanalysis projects.
Key changes:
- Implemented netCDF file reading with dimension and variable extraction in the C++ IMS reader
- Changed the staged file extension from
.ascto.datto accommodate both ASCII and netCDF formats - Maintained backward compatibility by attempting netCDF first, then falling back to ASCII if that fails
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| utils/land/gdas_fv3jedi_calc_scf_to_ioda.cc | Implements netCDF reading logic including dimension retrieval, variable extraction, and data buffer processing for IMS surface values |
| parm/snow/snow_stage_ims_scf2ioda.yaml.j2 | Updates file extension from .asc to .dat to support both ASCII and netCDF formats |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Thanks @jiaruidong2017. have you confirmed that we get the same output with a netcdf and an ascii file? They should be identical. |
I just made a comparison for one time and confirmed that the outputs are identical by using a netcdf file and an ascii file. |
Great - thanks. I'll approve now. |
ClaraDraper-NOAA
left a comment
ClaraDraper-NOAA
left a comment
There was a problem hiding this comment.
Approving, subject to adding the checks requested by co-pilot (where appropriate).
CoryMartin-NOAA
left a comment
CoryMartin-NOAA
left a comment
There was a problem hiding this comment.
C++ looks good but I think we should be more explicit with the file suffixes
Update the input file suffixes from `.asc` to `.dat`, including those for ASCII and NetCDF formats. Contributes to NOAA-EMC/GDASApp#2023 --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
|
@jiaruidong2017 update the jcb-gdas hash to the head of develop and then I can get this approved and merged. Thanks! |
|
@CoryMartin-NOAA Thanks and update the jcb-gdas repo |
CoryMartin-NOAA
left a comment
CoryMartin-NOAA
left a comment
There was a problem hiding this comment.
I didn't test but it looks good from what I can tell, thanks Jiarui!
Update the input file suffixes from `.asc` to `.dat`, including those for ASCII and NetCDF formats. Contributes to #2023 --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
# Description The current C++ IMS preprocessing code supports only ASCII-format input files. This PR adds support for netCDF input files to meet the requirements of reanalysis projects. Contributes to NOAA-EMC/global-workflow#4387 # Companion PRs NOAA-EMC/jcb-gdas#221 # Issues Resolves #2022 # Automated CI tests to run in Global Workflow <!-- Which Global Workflow CI tests are required to adequately test this PR? --> - [ ] atm_jjob <!-- JEDI atm single cycle DA !--> - [ ] C96C48_ufs_hybatmDA <!-- JEDI atm cycled DA !--> - [ ] C96C48_hybatmsnowDA <!-- JEDI snow cycled DA !--> - [ ] C96_gcafs_cycled <!-- JEDI aerosol cycled DA !--> - [ ] C48mx500_3DVarAOWCDA <!-- JEDI low-res marine 3DVar cycled DA !--> - [ ] C48mx500_hybAOWCDA <!-- JEDI marine hybrid envar cycled DA !--> - [ ] C96C48_ufsgsi_hybatmDA <!-- JEDI atm Var with GSI EnKF cycled DA !--> - [ ] C96C48_hybatmDA <!-- GSI atm cycled DA !-->
Description
The current C++ IMS preprocessing code supports only ASCII-format input files. This PR adds support for netCDF input files to meet the requirements of reanalysis projects.
Contributes to NOAA-EMC/global-workflow#4387
Companion PRs
NOAA-EMC/jcb-gdas#221
Issues
Resolves #2022
Automated CI tests to run in Global Workflow