Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generic seralization api needs to be reworked #253

Open
cyrush opened this issue Feb 19, 2025 · 1 comment
Open

generic seralization api needs to be reworked #253

cyrush opened this issue Feb 19, 2025 · 1 comment
Labels

Comments

@cyrush
Copy link
Member

cyrush commented Feb 19, 2025

To serialize / deserialize, the docs show we need to declare a zfp::array::header instance, however it is now an abstract class:

'zfp::array::header' is an abstract class
zfp::array::header header;

https://zfp.readthedocs.io/en/release1.0.0/arrays.html#serialization-1

For 1.0, the ZFP api changed to use concrete dimensioned types -- and this change undermined the generic seralization cases (using base zfp::array::header and zfp::array)

@lindstro
Copy link
Member

As discussed offline, (de)serialization indeed appears broken. A possible workaround for serialization of a concrete array arr of type arr_type (e.g., zfp::array3<double>) is to use the derived type arr_type::header as header type, or (in C++11) decltype(arr)::header. If arr_type is the base class zfp::array, one likely would need a switch statement on arr.scalar_type() and arr.dimensionality() to determine the correct header type.

Deserialization is more challenging, as it requires first constructing a header object from a byte buffer; see constructor. However, the size of that buffer depends in principle on the header type being encoded in the buffer, which creates a chicken-and-egg problem. This is potentially a lesser problem if the header and compressed data have already been read into memory together, e.g., with the header immediately preceding the compressed data (as suggested by the serialization documentation).

It should be pointed out that the zfp CODEC header constructor performs various checks, e.g., to ensure that the optional bytes argument (i.e., the buffer size) matches the encoded header length and that the buffer stores a well-formed zfp header. If neither is true, an exception is raised. Upon successful construction, one can also interrogate the header for scalar type and dimensionality to choose a matching array class, e.g., in a switch statement.

This certainly is not what we had in mind when we redesigned zfp (de)serialization. This is a bug that will be addressed in a future release.

@lindstro lindstro added the bug label Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants