Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a bidirectional coder for Protocol Buffers' Text Format #130

Open
joshsh opened this issue Aug 9, 2024 · 0 comments
Open

Add a bidirectional coder for Protocol Buffers' Text Format #130

joshsh opened this issue Aug 9, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@joshsh
Copy link
Collaborator

joshsh commented Aug 9, 2024

For practical reasons, Hydra needs to integrate more deeply with Protocol Buffers. It has become common to define a model in Hydra (for the sake of a richer language with better support for transformations, domain-specific validation, etc.) and use Hydra's Protobuf coder to also materialize the model in Protobuf. However, passing data between code generated from the Protobuf, and code generated directly from the Hydra model, has necessitated hand-written mapping functions for each and every type. These functions are time-consuming to write, although tools like GitHub Copilot can cut down the tedium by at least half. We need a better way.

Once #47 is done, the better way will be to write a bidirectional coder to translate between Hydra typed term representations, and one of Protobuf's built-in serialization formats (an alternative would be to write code to generate Java mappers for a given model, or Go mappers, etc., but this would need to be done on a language-by-language basis due to differences in the Protobuf APIs in each language). The contenders are:

  • Binary Format -- most compact and efficient, and universally supported
  • JSON Format -- would be convenient due to existing support for JSON in Hydra. Unfortunately, there is uneven support for Protobuf JSON across language-specific implementations. E.g. support in Go is pretty poor.
  • Text Format -- a human-readable format which is well supported in both the Java and Go implementations of Protobuf, and presumably in a number of other languages

I propose to start with the Text Format for the sake of ease of development, debuggability. We can proceed to the binary format if that goes well. Note that a new Hydra model for the Text Format will also be required.

@joshsh joshsh added the enhancement New feature or request label Aug 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

No branches or pull requests

1 participant