Serialization in Runtime components (RuntimeScene, RuntimeObject etc)

[arthuro555]

Description

Sometimes, you want to get all the data of an object to save a snapshot of your scenes state (saving data) or in my case get it to synchronize it between users in an online multiplayer game context. It is already possible but tidious and has to be adapted for each object type.

Solution suggested

Make a serializer and deserializer method to export and restore the pertinent data from an object. Then make an extension with actions to call those serializer on the object and stores them in variables and get such data from variables and restore it to an object.

Alternatives considered

As I said this is already possible but tidious and not very cross gd version compatible as you might want to store some internal values and do it through Js and if one property changes it's screwed while a serializer would automatically change but would still return old versions compatible serialized data.

[arthuro555]

See #1376

[arthuro555]

@4ian any opinion on this? Sorry for mentioning you but if I don't I'm afraid this issue and PR will stay open and unanswered for eternity

[4ian]

I think there are additional challenges when it comes to serializing objects:

• If used for networking, using a format as lightweight as possible, possibly serializing to a pack of bytes with a well known schema rather than JSON which has a lot of overhead (plain text vs binary, names of properties, etc...)
• Speaking of networking, should we allow some fields to be not serialized if we know we won't use them?
• Serialization itself may work, but then should there be some backward/forward compatibility? How do you handle unserializing from a previous version? How much versions do you support?
• When do you unserialize? Do you expect resources to be loaded already? In the futur, when GDevelop will load only textures from the current scene, what if you try to unserialize some object for which textures are not loaded?

The answer may range to very simple to very complicated for each question :) All of this depends on the use case: what is this serialization intended for (save/load game? Some naive json will do the trick! Networking? Now you have to handle proper packing, super fast [un]serialization, etc...).

There is also the cost in terms of code size and maintenance for adding [un]serialization methods to all objects:

• this will increase the code size for of the engine, are we ok with this?
• this will need developers to do extra work for all the objects they are doing. Should we instead think of a semi automated way of generating this serialization? Should we do this as something that is optional: for example a function unserialize/serialize that explicitly show that there is support for serialization. If they are missing, these objects are marked as not supporting serialization.
• can we do this as an extension instead? Though that's a bit "weird" because this extension will implement things for other extensions, so there is some careful design to be chosen.

Again all of these can have simple or complex answers, but I would need some careful explanations of all of this, explaining why some decisions are made, how can we do something that is both performant/fast/lightweight and requiring as few input from developers as possible (everything may not be possible of course ;)).

---

Side note: for a long time, GDevelop has been packed with tons of features for the sake of making features. Turns out that these features were poorly implemented and worse, nobody would use them! "Less is more" => I prefer to avoid something unless we are confident about: 1) users wanting this and will use this 2) we have a great implementation in mind.

[arthuro555]

Actually, I used a completely wrong term. This is not a complete serialization, just something to get all the relevant data of an object to then serialize it. Because people don't want to go read the source or study the internal (or not) variables specific to each objects, or go get them individually. This is a kind of way for the developper to choose what is exported or not. Because you cannot serialize an object directly. For the texture loading, this is not one of the things I meant. What will be saved is not the texture but what animation and frame it is at. This is made to be applied on the same object it was made from. For version cross compatibility, I think following strategies are good: 1. Keeping the "serializer version" (bumped when changing an internal variables name, adding or removing one) on the serialized data (actually I will call it dumped data now as it seems more adequate) 2. Naming under both the new and old name a variable in the dumped data 3. Use of a placeholder for deleted variables 4. Keeping a "last compatible serializer version" value in the dumped data to let old versions know if they are compatible 5. Making new serializer retro compatible The problem would be that it would become huge if there are many modifications (Wich shouldn't happen but we never know) so Devs should be warned that the old values are deprecated and remove them after 6 month of deprecation. For the code size, I agree that it takes some place, but really, is it that much code? I mean, just by cleaning up objects and behavior registration I think it saved approximately what we would loose, for making for example saving and networking easier I think it's worth it. For the extension I was talking about, I meant something like JSON because it is easy to understand, use and debug for GDevelop users. The problem for the networking extension is that we need something lightweight BUT available in many languages. As the ones who actually use it need to code their server and therefore know how to code, I think we can let them do however they want with JS. The extension I was talking about is really for saving. For the extra work for developpers, it's not that much in my opinion and in the worst case someone can do it for the original extension's Dev. A marking system would make no sense as marking as serialization is as much difficult as adding it.

…

________________________________ De : Florian Rival <[email protected]> Envoyé : samedi 18 janvier 2020 à 13:01 À : 4ian/GDevelop Cc : Arthur Pacaud; Author Objet : Re: [4ian/GDevelop] Serialization in Runtime components (RuntimeScene, RuntimeObject etc) (#1381) I think there are additional challenges when it comes to serialize objects: * If used for networking, using a format as lightweight as possible, possibly serializing to a pack of bytes with a well known schema rather than JSON which has a lot of overhead (plain text vs binary, names of properties, etc...) * Speaking of networking, should we allow some fields to be not serialized if we know we won't use them? * Serialization itself may work, but then should there be some backward/forward compatibility? How do you handle unserializing from a previous version? How much versions do you support? * When do you unserialize? Do you expect resources to be loaded already? In the futur, when GDevelop will load only textures from the current scene, what if you try to unserialize some object for which textures are not loaded? The answer may range to very simple to very complicated for each question :) All of this depends on the use case: what is this serialization intended for (save/load game? Some naive json will do the trick! Networking? Now you have to handle proper packing, super fast [un]serialization, etc...). There is also the cost in terms of code size and maintenance for adding [un]serialization methods to all objects: * this will increase the code size for of the engine, are we ok with this? * this will need developers to do extra work for all the objects they are doing. Should we instead think of a semi automated way of generating this serialization? Should we do this as something that is optional: for example a function unserialize/serialize that explicitly show that there is support for serialization. If they are missing, these objects are marked as not supporting serialization. * can we do this as an extension instead? Though that's a bit "weird" because this extension will implement things for other extensions, so there is some careful design to be chosen. Again all of these can have simple or complex answers, but I would need some careful explanations of all of this, explaining why some decisions are made, how can we do something that is both performant/fast/lightweight and requiring as few input from developers as possible (everything may not be possible of course ;)). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#1381?email_source=notifications&email_token=AETT4LQYHFMGJASCZPA4HVTQ6LVP7A5CNFSM4KHOUBSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJJWYOY#issuecomment-575892539>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AETT4LXJQRVLEDHCXLM6YKDQ6LVP7ANCNFSM4KHOUBSA>.

[4ian]

I mean, just by cleaning up objects and behavior registration I think it saved approximately what we would loose, for making for example saving and networking easier I think it's worth it.

Careful with this kind of argument, the PR you're talking about is adding 54 lines, removing 102, so a net benefit of 48 lines - which is good! But nothing comparable to 1 serialization + 1 unserialization method multiplied by the number of objects used in the game.
Anyway, not a huge deal, especially if we can avoid including these serialization methods if they are not used (for example, by including in external .serialization.js files, or something like this).

I still think that we're discussing something that is technical and can be risky in terms of maintenance/performance/additional burden for people that want to contribute. So we must get it right. And to get something right, we must not start from the technical solutions but from the user needs :) What is the thing that we're trying to implement? Networking? Save/Load? Both?

Would be great to have an idea of what of these, or something else, is more important - unless there is already a thread on the forum that I've not seen? :)

[4ian]

Note that while I'm complaining how use cases, it's just because I want to avoid losing time or going in a wrong technical direction. You're free to try some prototyping to get a better idea of how this is working, and if this works for something like save/load and/or networking, or something else ☺️

[arthuro555]

Well, my bad for the PR argument, you're totally right. I think I get your point of making serializer components in a separate extension. It's a good idea. It's weird but would let a possibility of not having the code in the end game. I will try that approach. Like I was saying this is more for saving as networking people should have programming knowledge to make the server side so they should also be able to implement their own thing in Js. What I will do is probably add a serializertools file with the serialization part separed from the "data dumping" si people can call the "data dumping" appart through Js.