feat: add miden-objects with canonical Protobuf representations by kkovaacs · Pull Request #3707 · 0xMiden/protocol

kkovaacs · 2026-08-24T14:59:24Z

Summary

Supports 0xMiden/node#2471 (0xMiden/node#2471).

The node’s gRPC migration replaces opaque, Miden-native byte payloads with structured Protobuf messages. These object schemas belong alongside the protocol definitions rather than inside an RPC implementation: keeping them in the node would force clients and services to duplicate generated types and protocol validation, allowing their representations to drift.

This PR introduces miden-objects as the canonical transport representation for protocol objects exchanged between clients and nodes. It does not replace miden-protocol’s native serialization or define RPC services.

Changes

Added structured schemas for field elements, words, accounts and account patches, notes and attachments, transactions and batches, block headers and bodies, partial blockchains, partial MMR, Merkle data, and signed blocks.
Added bidirectional conversions between generated messages and protocol types, with field-path-aware errors and validation delegated to protocol constructors from feat(protocol): validate batch and block data at construction #3706.
Kept execution proofs, MAST forests, public keys, and signatures in their canonical encodings behind typed Protobuf wrapper messages, avoiding raw bytes fields throughout higher-level messages.
Exposed a self-contained descriptor set and canonical Prost external paths so RPC crates can import these schemas without generating duplicate Rust object types.

Existing protocol serialization and commitment calculation are unchanged by this PR.

PhilippGackstatter · 2026-08-25T08:33:18Z

+message AccountHeader {
+  AccountId account_id = 1;
+  primitives.Digest vault_root = 2;
+  primitives.Digest storage_commitment = 3;
+  primitives.Digest code_commitment = 4;
+  uint64 nonce = 5;
+}


Would this also need to have the AccountHeader::VERSION_1? It is private for now, but we can make it public, or even make it a field of the Rust struct AccountHeader it it helps.

If so, the same probably applies to PartialNoteMetadata, AssetId and BlockHeader.

What's the use case for the version? Is it accounted for in the commitment?

Protobuf messages can be extended with new fields without breaking compatibility, so some changes can be modeled in Protobuf without having that explicit version number.

Semantic changes to these types might require a more complicated approach though. Even in that case I don't think adding a version field to this message would make sense. I'd rather do something like this instead:

message AccountHeader { oneof version { AccountHeaderV0 v0 = 1; AccountHeaderV1 v1 = 2; } } message AccountHeaderV0 { // exact representation of V0 } message AccountHeaderV1 { // exact representation of V1 }

What's the use case for the version? Is it accounted for in the commitment?

The purpose is to indicate changes to the structure (what protobuf seems to handle already). Presumably only field additions, but can't say for sure. It's really hard to predict how things will evolve. Basically, I don't know of any planned changes to the account header, but that doesn't mean there won't ever be any.

The version is committed to, yes.

@Mirko-von-Leipzig do you think the oneof above would be better here? It's definitely more future-proof but adds some complexity to the protocol.

I would almost always opt for oneof. Doubly-so if we are uncertain about what kind of change we are introducing.

One alternative is that we can separate the account struct version from the schema version, but that seems like a bad idea.

One thing I do want to clarify is that we are not versioning just the account header here - we are versioning a full account (this is why I think PartialAccount should also get a version). This is a bit more clear for NoteMetadata - there too we are versioning the full note - not just the metadata field. So, if we do want to keep the oneof, it would have to be at the Note level rather than the NoteMetadata level.

Yep, that's a good point: top-level objects should be versioned. That also means that in the case of notes we should version both Note and NoteHeader (because we're using NoteHeader for output notes in transactions).

That's probably the best argument against the oneof representation?

I think NoteHeader is versioned implicitly because it contains NoteMetadata. But yeah, expressing this relationship with oneof may be challenging.

I think my fear with the version field is that the flat message gradually becomes a union of every version’s fields. Requiredness and valid field combinations then depend on the version, so the schema itself permits invalid cross-version combinations.

Conversion would also need to support practically all versions if we're to use the protobuf representations in long-term storage.

Yes, agreed that this is a concern. My main hope is that the number of versions will be pretty small and we won't end up with too many invalid combinations (if any). And if we do end up with some, we'd be able to handle them at conversion time.

OK, I've removed the oneof messages and have added version fields instead.

PhilippGackstatter

Only a partial review so far, but left a few suggestions. Overall looking good!

PhilippGackstatter · 2026-08-25T13:38:51Z

+    fn from(value: &AccountWitness) -> Self {
+        Self {
+            account_id: Some(value.id().into()),
+            witness_id: Some(value.id().into()),
+            commitment: Some(value.state_commitment().into()),
+            path: Some(value.path().clone().into()),


nit: could we replace all the values with more precise names, e.g. witness? This makes things easier to read and an LLM does it in no time.

bobbinth

Looks good! Thank you! I mostly focused on reviewing protobuf code and skimmed over most other code. I left some comments inline - some may require alignment but most are pretty straight forward (I think).

bobbinth

I mostly reviewed the protobuf files - but looks good! Thank you!

The tests are failing now - I suspect this may be due to the v0.30 VM migration.

kkovaacs · 2026-09-03T07:25:03Z

I mostly reviewed the protobuf files - but looks good! Thank you!

The tests are failing now - I suspect this may be due to the v0.30 VM migration.

Yep, I've merged next and fixed the 0.30 VM migration fallout.

bobbinth

I mostly double-checked protobuf changes again - all looks good there! Thank you!

bobbinth · 2026-09-03T09:04:25Z

Do we not need a ProvenBlock (or something like that) message? If we do - I'd add it in a follow-up PR (rather than this one).

bobbinth · 2026-09-03T09:07:24Z

+message PublicKey {
+  bytes encoded = 1;
+}


We'll most likely have EdDSA and ECDSA over P256 curve soon - but all of these will be just variant + the actual key - so, the message structure could remain the same.

zeapoz

Looks good to me! I left two suggestion that I think are worth considering before merging, but I wouldn't consider them blocking

zeapoz · 2026-09-03T10:57:59Z

+// STORAGE PATCHES
+// ================================================================================================
+
+const fn encode_storage_operation(operation: StoragePatchOperation) -> i32 {


What do you think about adding a small doc section to these mappping functions just to mention the fact that the integer representation is different between the domain types and the proto types? As an example, here we have:

Rust StoragePatchOperation (patch_operation.rs):

Variant u8

Create 0

Update 1

Remove 2

Proto StoragePatchOperation (account.proto):

Variant Proto value

STORAGE_PATCH_OPERATION_UNSPECIFIED 0

STORAGE_PATCH_OPERATION_CREATE 1

STORAGE_PATCH_OPERATION_UPDATE 2

STORAGE_PATCH_OPERATION_REMOVE 3

Since best practice for protobuf involve storing an explicit unspecified operation all other values shift by one, so I would suggest mentioning that the encoding is slightly different in the docs

These are the ones affected that I found:
StoragePatchOperation
AssetComposition
StorageSlotType
NoteType

kkovaacs force-pushed the krisztian/protobuf branch 2 times, most recently from 6114646 to 0e98371 Compare August 24, 2026 15:26

kkovaacs changed the title ~~krisztian/protobuf~~ feat: add miden-objects with canonical Protobuf representations Aug 24, 2026

kkovaacs force-pushed the krisztian/protobuf branch from 0e98371 to 5db9107 Compare August 24, 2026 15:41

kkovaacs requested a review from PhilippGackstatter August 24, 2026 15:49

kkovaacs marked this pull request as ready for review August 24, 2026 15:50

kkovaacs requested a review from Mirko-von-Leipzig August 24, 2026 15:51

Mirko-von-Leipzig reviewed Aug 24, 2026

View reviewed changes

PhilippGackstatter reviewed Aug 25, 2026

View reviewed changes

kkovaacs force-pushed the krisztian/protobuf branch from e436244 to 7b51099 Compare August 26, 2026 11:36

PhilippGackstatter mentioned this pull request Aug 26, 2026

feat: introduce ProtocolConfig, KernelConfig and ValidatorConfig #3725

Merged

kkovaacs force-pushed the krisztian/protobuf branch from 7b51099 to bd7e76f Compare August 28, 2026 14:43

Base automatically changed from krisztian/add-validated-constructors to next August 29, 2026 21:10

kkovaacs force-pushed the krisztian/protobuf branch from 4114482 to 05183a3 Compare August 29, 2026 21:10

kkovaacs added 15 commits August 31, 2026 08:29

feat: add miden-objects crate with canonical Protobuf encodings

e6a3656

refactor(proto): reshuffle protobuf files

bcc857e

fix: add ExecutionProof wrapper

6eb02c1

fix: add MastForest wrapper

89597d2

fix: cargo shear

f93a28b

fix: extract generic PublicKey

334db64

fix: extract generic Signature

9a76aac

fix: remove redundant checks from conversion code

0f30528

fix: fix conversion compile errors after rebase

bbfde6a

chore: update CHANGELOG

e90344c

fix: fix compile errors after rebase

7cb15d6

chore: allow duplicate protobuf build dependencies

ba87bf7

chore: remove schema tests

a7a01c0

fix(objects): adapt conversions to validated constructors

60c3515

refactor(objects): consolidate protobuf primitive types

0a6aec4

kkovaacs added 2 commits September 1, 2026 22:21

refactor: clarify feature check package list

a599605

fix(objects): preserve note attachments error context

ed5210d

bobbinth reviewed Sep 1, 2026

View reviewed changes

kkovaacs added 10 commits September 2, 2026 12:33

refactor(objects): clarify protobuf primitive variants

a95c230

refactor(objects): structure protobuf asset identifiers

5e5c6ec

refactor(objects): align protobuf note transactions

3e09192

fix(objects): remove redundant asset test clones

0dea066

fix(objects): remove legacy protobuf asset schema

3bd0a01

refactor(objects): add protobuf account versions

12a6830

refactor(objects): flatten protobuf block version

930d2cb

refactor(objects): flatten protobuf note version

9081c81

refactor(objects): add protobuf asset version

6af016f

Merge remote-tracking branch 'origin/next' into krisztian/protobuf

1b16fd7

bobbinth approved these changes Sep 2, 2026

View reviewed changes

kkovaacs added 5 commits September 3, 2026 08:57

fix(objects): adapt partial SMT protobuf to miden-crypto 0.30

354d3ca

fix(objects): update execution proof test fixtures

cf1be6e

fix(objects): update partial SMT error assertions

78cb363

style(objects): format partial SMT imports

d3ecd67

fix(objects): preserve partial SMT depth boundaries

e679e29

bobbinth approved these changes Sep 3, 2026

View reviewed changes

kkovaacs added 2 commits September 3, 2026 12:14

Merge remote-tracking branch 'origin/next' into krisztian/protobuf

2e33f42

Merge remote-tracking branch 'origin/next' into krisztian/protobuf

f4c965d

zeapoz approved these changes Sep 3, 2026

View reviewed changes

Mirko-von-Leipzig approved these changes Sep 3, 2026

View reviewed changes

fix(objects): keep note type enum consistent with upstream type

78985ea

kkovaacs added this pull request to the merge queue Sep 3, 2026

Merged via the queue into next with commit de00863 Sep 3, 2026
19 checks passed

kkovaacs deleted the krisztian/protobuf branch September 3, 2026 11:57

Variant	Proto value
`STORAGE_PATCH_OPERATION_UNSPECIFIED`	`0`
`STORAGE_PATCH_OPERATION_CREATE`	`1`
`STORAGE_PATCH_OPERATION_UPDATE`	`2`
`STORAGE_PATCH_OPERATION_REMOVE`	`3`

Uh oh!

Conversation

kkovaacs commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PhilippGackstatter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bobbinth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bobbinth left a comment

Choose a reason for hiding this comment

Uh oh!

kkovaacs commented Sep 3, 2026

Uh oh!

bobbinth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeapoz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kkovaacs commented Aug 24, 2026 •

edited

Loading