Skip to content

New C++ NetCDF I/O interface is missing compression #65

@st-ncar

Description

@st-ncar

The new C++ NetCDF I/O interface does not support data compression yet, which leads to significantly larger file sizes. Consider, for example, a IODA file for MHS data. The file size resulting from the C++ NetCDF I/O is

$ du -sh mhs_n18_obs_2018041500.h5
66M	mhs_n18_obs_2018041500.h5

while the file size resulting from the old Fortran NetCDF I/O is

$ du -sh mhs_n18_obs_2018041500.h5
20M	mhs_n18_obs_2018041500.h5

A comparison of the variable encoding settings suggests that compression is not enabled when generating files with the new C++ interface

$ nccmp -lef file_new_interface.h5 file_old_interface.h5
[...]
DIFFER : VARIABLE : dateTime : CHUNK STORAGE : NC_CONTIGUOUS <> NC_CHUNKED
DIFFER : VARIABLE : dateTime : CHUNK SIZES : 0 <> 5645
DIFFER : VARIABLE : dateTime : DEFLATE : 0 <> 1
DIFFER : VARIABLE : dateTime : DEFLATE LEVEL : 0 <> 6
[...]

We should enable variable compression in the C++ NetCDF I/O interface to minimize disk usage. The NetCDF library should support variable compression out-of-the box, it is probably just a matter of adding the right parameters / function calls.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions