Limitations of serialization implementation regarding shared references

[ratmice]

With the added serde derives, in PR #26 it came up on zulip that the serialization of images is "not ideal", but useful enough to keep for now but might eventually require a breaking change.

The general issue is that Image Blobs may be shared, which when serialized will duplicate the data.
This then deserializes to multiple images each having their own unique ID.

serde

The most relevant issue I found in serde is the following involving the serde_state crate
which adds on top of serde SerializeState and DeserializeState traits, alas this is an experimental fork of serde
which is not in the maintenance goldilocks zone at 110 commits ahead and 745 commits behind serde itself.

serde-rs/serde#881

I guess my recommendation for the current state would be if serializing shared images serialize them to a map/vector
and reference those by key or index.

Alternatives to serde

Rkyv by default does the right thing for serializing shared pointers
It isn't really comparable to serde though in that it is a fixed data format rather than a data model supporting multiple data formats.

I think it might be more ideal, because it would be both zero copy, and we don't have to work around the data model to avoid duplicating shared references.

There is one issue which gives me some pause, that I would probably want to fix/work on before integrating it rkyv/rkyv#285 because rkyv serializes enum variants by their representation value rather than nominally. Simply reordering enums or adding variants that aren't appended to the current variants can cause compatibility hazards.

[ratmice]

For my thoughts/usage of this I'm not really leveraging the need for serde's support for multiple data formats, and rkyv support would be sufficient. So if that is something that seems agreeable/worthwhile i'd be happy to work towards that, and moving most of my serde usage over to it.