Serializer/deserializer for hierarchical, multi-type data

[tigercosmos]

Implement a serializer/deserializer for hierarchical data of multiple types.

Currently the implementation of CallProfilerSerializer::serialize_caller_profile in 1f03e8e just hardcoded the serialization logic, which could be replaced by a generic JSON serializer.

Progress:

• General Serializer/Deserializer Interface
• JSON port
• YAML port (Low priority)
• Python port using pybind11

[yungyuc]

@tigercosmos Please specify how the general serializer should look like, including supporting the existing JSON serialization in Toggle:

modmesh/tests/test_toggle.py

Line 305 in 1f03e8e

def test_to_json(self):

This may use more discussions. Consider to create a discussion conversion or turn this issue into one.

[tigercosmos]

@yungyuc I like the way Golang does JSON marshaling and unmarshalling, not sure if it is possible to make it in C++.

type Person struct {
    Name   string `json:"name"`
    Age    int    `json:"age"`
    Email  string `json:"email"`
}

func main() {
    person := Person{
        Name:  "Jane Doe",
        Age:   25,
        Email: "[email protected]",
    }

    // Marshal the person struct into JSON data
    jsonData, err := json.Marshal(person)
    if err != nil {
        fmt.Println("Error marshaling to JSON:", err)
        return
    }

    // Print the JSON data
    fmt.Println(string(jsonData))
}

type Person struct {
    Name   string `json:"name"`
    Age    int    `json:"age"`
    Email  string `json:"email"`
}

func main() {
    jsonData := `{"name": "John Doe", "age": 30, "email": "[email protected]"}`

    var person Person

    // Unmarshal the JSON data into the person struct
    err := json.Unmarshal([]byte(jsonData), &person)
    if err != nil {
        fmt.Println("Error unmarshaling JSON:", err)
        return
    }

    // Print the unmarshaled struct
    fmt.Printf("%+v\n", person)
}

[yungyuc]

I do not have a preference and we can follow yours, @tigercosmos . Please author a mock-up example in C++. You can add it in a comment like what you did for golang.

[tigercosmos]

@yungyuc The following is the proposal for the design. There should be no problem applying the design on Toggle or CallProfilerSerializer

The interface:

class JsonSerializable {
public:
    virtual std::string to_json() const = 0;
    virtual void from_json(const std::string& json) = 0;
};

The object inherits the JsonSerializable interface:

class Person : public JsonSerializable {
    std::string name;
    int age;
    std::string email;
    Address address;
    std::vector<std::string> phoneNumbers;

public:
    // macro (to be implemented), should be able to support array, object, nested structure
    // also implement `to_json` and `from_json` in the macro.
    JSON_SERIALIZABLE(
        register("name", name);
        register("age", age);
        register("email", email);
        register("address", address);
        register("phoneNumbers", phoneNumbers);
    )
};

Usage:

Person person(...);
std::string jsonData = person.to_json();
std::cout << jsonData << std::endl; // {"name":"Bob","age":25,"email":"[email protected]", .... }

Person newPerson;
newPerson.from_json(jsonData);

[yungyuc]

@tigercosmos this looks nice! I wonder if it needs to be limited to JSON? The API seems to be applicable to any hierarchical container like YAML and Python dictionary. If that is true, maybe we can rename it from JsonSerializable to SerializableItem. What do you think?

[tigercosmos]

@yungyuc yes, you are correct. for now, let's do the json first, and we can have yaml and other later.

[yungyuc]

Great. When do you plan to have a prototype or initial implementation for discussion, @tigercosmos ?

But I think at this stage we only need JSON I/O. YAML, Python dictionary (pickle), and other data formats can be planned as future work.

[tigercosmos]

I can do the JSON part soon, it shouldn't take too much time.

[yungyuc]

@tigercosmos Does any work remain with this issue?

[tigercosmos]

Serialization for Python object using pybind11 can be the next step.

[yungyuc]

Serialization for Python object using pybind11 can be the next step.

Sounds good. Could you please update the progress in the beginning of the issue?

[tigercosmos]

It's updated now.