[Epic] Tofu -- Rust to other languages mapping tool

[MaksymZavershynskyi]

As a continuation of rpctypegen introduced in #1854 we should have a tool that allows us to map any Rust struct/enum/union to its closest equivalent in a different language.
The tool should be implemented as a cargo plugin and will use syn and quote to process the source code of the given project.

For example, the following command will iterate over the file structure of the given Rust project and for each file generate Python file that for each Rust struct/enum/union struct contains Python class together with methods that allow us to serialize/deserialize Rust struct/enum/union <-> Python class between each other using JSON or Borsh:

cargo rusttofu --lang=py --output=/tmp/my_python_project

Note, that currently for some languages we are using an intermediate JSON schema (e.g. for TypeScript and Python) to perform serialization/deserialization. With this tool, it might not be needed, since we can generate the TypeScript or Python code.

We would need to support the following languages for which we are already doing the mapping manually: C/TypeScript/AssemblyScript/Python/Solidity .

Note, I propose the name Rust Tofu as a play on the phrase "Rust to foo" where "foo" denotes something. Other name suggestions are welcome.

[vgrichina]

Note, that currently for some languages we are using an intermediate JSON schema (e.g. for TypeScript and Python) to perform serialization/deserialization. With this tool, it might not be needed, since we can generate the TypeScript or Python code.

I think it's good to have intermediate JSON schema to make it easy to use in any language (even ones we don't support) without having to deal with Rust. E.g. if I'm developing iOS app using SDK on Swift – I likely don't want to deal with Rust stuff.

[MaksymZavershynskyi]

Note, that currently for some languages we are using an intermediate JSON schema (e.g. for TypeScript and Python) to perform serialization/deserialization. With this tool, it might not be needed, since we can generate the TypeScript or Python code.

I think it's good to have intermediate JSON schema to make it easy to use in any language (even ones we don't support) without having to deal with Rust. E.g. if I'm developing iOS app using SDK on Swift – I likely don't want to deal with Rust stuff.

We can keep it for some languages. It obviously won't work for languages like C.

[mfornet]

I think it's good to have intermediate JSON schema to make it easy to use in any language (even ones we don't support) without having to deal with Rust. E.g. if I'm developing iOS app using SDK on Swift – I likely don't want to deal with Rust stuff.

Json can be one of the output "languages".

[mfornet]

We can keep it for some languages. It obviously won't work for languages like C.

Btw, where are we using this for C right now?

[MaksymZavershynskyi]

Note, that currently for some languages we are using an intermediate JSON schema (e.g. for TypeScript and Python) to perform serialization/deserialization. With this tool, it might not be needed, since we can generate the TypeScript or Python code.

I think it's good to have intermediate JSON schema to make it easy to use in any language (even ones we don't support) without having to deal with Rust. E.g. if I'm developing iOS app using SDK on Swift – I likely don't want to deal with Rust stuff.

We can keep it for some languages. It obviously won't work for languages like C.

Btw, where are we using this for C right now?

https://www.ledger.com/

[vgrichina]

@nearmax I mean more that whoever would be maintaining generator for given language likely will prefer to do it in their language vs Rust. Having schema as JSON is useful for that.

[ailisp]

with methods that allow us to serialize/deserialize Rust struct/enum/union <-> Python class between each other using JSON or Borsh:

Would it be good to make this part of borsh project?

[ailisp]

Having schema as JSON is useful for that.

this sounds a good alternative too? but existing rust code need to be refactored to use structs generated from json schema?

[vgrichina]

@nearmax @mfornet parsing Borsh on Ledger is completely hardcoded. It's hard to do anything generally useful (beyond few helper functions) for embedded C as it's way too limited in RAM, so you typically specify every output to write to UI structs directly.

On C++ side (TrustWallet) I just needed to serialize which was also pretty easy to handcode using helper functions. It had to convert from protos (used internally) anyway, so again wouldn't be much benefit from having generated classes for Borsh (vs helper functions to write them out).

[MaksymZavershynskyi]

with methods that allow us to serialize/deserialize Rust struct/enum/union <-> Python class between each other using JSON or Borsh:

Would it be good to make this part of borsh project?

I would make it generally compatible with various serializers, we can start with Borsh and JSON. External contributors can easily add CBOR since it already supports various languages. Therefore I wouldn't make it a part of borsh.io.

[MaksymZavershynskyi]

On C++ side (TrustWallet) I just needed to serialize which was also pretty easy to handcode using helper functions. It had to convert from protos (used internally) anyway, so again wouldn't be much benefit from having generated classes for Borsh (vs helper functions to write them out).

The benefit is in the consistency and making sure we are not making mistakes in the process of copy-paste. There are multiple ways one can map between Rust and C/C++ structures/enums/unions. The earlier we standardize it the better for our future API. Also looking back at how many Rust structures we expose through RPC I am wary of someone making a mistake when manually writing out their counterparts in a different language.

@nearmax @mfornet parsing Borsh on Ledger is completely hardcoded.

There might be a miscommunication on what this tool does. It is not a library. It generates C/C++ structures that are matching Rust structures that you would otherwise write out manually.

[mfornet]

Note, that currently for some languages we are using an intermediate JSON schema (e.g. for TypeScript and Python) to perform serialization/deserialization. With this tool, it might not be needed, since we can generate the TypeScript or Python code.

I think it's good to have intermediate JSON schema to make it easy to use in any language (even ones we don't support) without having to deal with Rust. E.g. if I'm developing iOS app using SDK on Swift – I likely don't want to deal with Rust stuff.

I understand now what @vgrichina is proposing and I think is a good idea. Instead of making this tool Rust-centric we should make it json-centric. We agree on some json-schema which I think we already did, and this tool just translate Foo => json-schema & json-schema => Foo.

Particularly in first tofu iteration we should only implement:
rust => json-schema &
json-schema => C/TypeScript/AssemblyScript/Python/Solidity

This way is much simpler to reason about, and it definitely scale better.

[MaksymZavershynskyi]

The schema does not need to be expressed in JSON, instead, it should be expressable in JSON.

Btw. Finding a language to describe types that interoperates with other languages nicely is a difficult task. Checkout witx as a solution to interface types in Wasm: https://www.youtube.com/watch?v=LCA9NnH7DxE its goal is to do exactly that.

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.

[bowenwang1996]

@nearmax what is the status of this issue? Should this even be an issue in nearcore?

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.