forked from ibanos90/obs2ioda
-
Notifications
You must be signed in to change notification settings - Fork 7
Open
Description
The new C++ NetCDF I/O interface does not support data compression yet, which leads to significantly larger file sizes. Consider, for example, a IODA file for MHS data. The file size resulting from the C++ NetCDF I/O is
$ du -sh mhs_n18_obs_2018041500.h5
66M mhs_n18_obs_2018041500.h5while the file size resulting from the old Fortran NetCDF I/O is
$ du -sh mhs_n18_obs_2018041500.h5
20M mhs_n18_obs_2018041500.h5A comparison of the variable encoding settings suggests that compression is not enabled when generating files with the new C++ interface
$ nccmp -lef file_new_interface.h5 file_old_interface.h5
[...]
DIFFER : VARIABLE : dateTime : CHUNK STORAGE : NC_CONTIGUOUS <> NC_CHUNKED
DIFFER : VARIABLE : dateTime : CHUNK SIZES : 0 <> 5645
DIFFER : VARIABLE : dateTime : DEFLATE : 0 <> 1
DIFFER : VARIABLE : dateTime : DEFLATE LEVEL : 0 <> 6
[...]We should enable variable compression in the C++ NetCDF I/O interface to minimize disk usage. The NetCDF library should support variable compression out-of-the box, it is probably just a matter of adding the right parameters / function calls.