Skip to content

Latest commit

 

History

History
366 lines (282 loc) · 15 KB

protobuf-idl.md

File metadata and controls

366 lines (282 loc) · 15 KB
title Protobuf IDL Support
sidebar_position 5
id protobuf-idl
license Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

This page explains how Apache Fory works with Protocol Buffers (.proto) schemas, how protobuf concepts map to Fory, and how to use protobuf-only Fory extension options.

What This Page Covers

  • Choosing protobuf vs Fory for your use case
  • Syntax and semantic differences that matter during adoption
  • Supported Fory extension options in protobuf files
  • Practical transition patterns from protobuf to Fory

Quick Decision Guide

Situation Recommended Format
You are building gRPC APIs and rely on protobuf tooling Protocol Buffers
You need maximum object-graph performance and ref tracking Fory
You need circular/shared references in serialized data Fory
You need strong unknown-field behavior for wire compatibility Protocol Buffers
You need native structs/classes instead of protobuf wrappers Fory

Protobuf vs Fory at a Glance

Aspect Protocol Buffers Fory
Primary purpose RPC/message contracts High-performance object serialization
Encoding model Tag-length-value Fory binary protocol
Reference tracking Not built-in First-class (ref)
Circular refs Not supported Supported
Unknown fields Preserved Not preserved
Generated types Protobuf-specific model types Native language constructs
gRPC ecosystem Native Java/Python/Go/Rust/C#/Dart/Scala/Kotlin/JavaScript service codegen

Fory can generate Java, Python, Go, Rust, C#, Dart, Scala, Kotlin, and JavaScript gRPC service companions with --grpc. JavaScript browser clients are generated with --grpc-web. Those services use normal gRPC transports but serialize request and response payloads with Fory rather than protobuf. For broad gRPC ecosystem tooling, schema reflection, and protobuf-native interceptors, protobuf remains the mature/default choice.

Why Use Apache Fory

  • Idiomatic generated code: Fory IDL generates language-idiomatic classes and structs that can be used directly as domain objects.
  • Faster serialization: In Fory benchmarks, Fory can be around 10x faster than protobuf for object serialization workloads.
  • Better graph modeling: Shared and circular references are first-class features instead of application-level ID-link workarounds.

See benchmark details under Performance References.

Syntax and Semantic Mapping

Package and File Options

Protocol Buffers

syntax = "proto3";
package example.models;
option java_package = "com.example.models";
option go_package = "example.com/models";

Fory

package example.models;

Fory uses one package namespace for cross-language registration. Language-specific package placement is still configurable in code generation.

Message and Enum Definitions

Protocol Buffers

message User {
  string id = 1;
  string name = 2;
  optional string email = 3;
  int32 age = 4;
  repeated string tags = 5;
  map<string, string> metadata = 6;
}

enum Status {
  STATUS_UNSPECIFIED = 0;
  STATUS_ACTIVE = 1;
}

Fory

message User [id=101] {
    string id = 1;
    string name = 2;
    optional string email = 3;
    int32 age = 4;
    list<string> tags = 5;
    map<string, string> metadata = 6;
}

enum Status [id=102] {
    UNKNOWN = 0;
    ACTIVE = 1;
}

Key differences:

  • Fory can assign stable type IDs directly ([id=...]).
  • Fory uses list<T> (with repeated T as alias).
  • Enum naming conventions are language-driven instead of protobuf prefix style.

oneof to union

Protobuf oneof is translated to a nested Fory union plus an optional field referencing that union.

Protocol Buffers

message Event {
  oneof payload {
    string text = 1;
    int32 number = 2;
  }
}

Fory-style shape after translation

message Event {
    union payload {
        string text = 1;
        int32 number = 2;
    }
    optional payload payload = 1;
}

Notes:

  • Union case IDs are derived from the original oneof field numbers.
  • The synthetic union field uses the smallest oneof case number.

Imports and Well-Known Types

Protobuf imports are supported. Common well-known types map directly:

  • google.protobuf.Timestamp -> timestamp
  • google.protobuf.Duration -> duration
  • google.protobuf.Any -> any

Type Mapping Highlights

Protobuf Type Fory Mapping
bool bool
int32, uint32 variable-length 32-bit integer kinds
sint32 zigzag 32-bit integer
int64, uint64 variable-length 64-bit integer kinds
sint64 zigzag 64-bit integer
fixed32, fixed64 fixed-width unsigned integer kinds
sfixed32, sfixed64 fixed-width signed integer kinds
float, double float32, float64
string, bytes string, bytes
repeated T list<T>
map<K, V> map<K, V>
optional T optional T
oneof union + optional union reference field
int64 [(fory).type = "tagged int64"] tagged int64 encoding
uint64 [(fory).type = "tagged uint64"] tagged uint64 encoding

Fory Extension Options (Protobuf)

Fory-specific options in .proto use the (fory). prefix.

option (fory).enable_auto_type_id = true;

message TreeNode {
  TreeNode parent = 1 [(fory).weak_ref = true];
  repeated TreeNode children = 2 [(fory).ref = true];
}

File-Level Options

Option Type Description
(fory).use_record_for_java_message bool Generate Java records for all messages in this file
(fory).polymorphism bool Enable polymorphic serialization metadata by default
(fory).enable_auto_type_id bool Auto-generate type IDs when omitted (compiler default is true)
(fory).evolving bool Default schema-evolution behavior for messages
(fory).go_nested_type_style string Go nested naming style: underscore (default) or camelcase
(fory).swift_namespace_style string Swift namespace style: enum (default) or flatten; applies only when package is non-empty

Message and Enum Options

Option Applies To Type Description
(fory).id message, enum int Explicit type ID for registration
(fory).alias message, enum string Alternate name used for auto-ID hashing
(fory).evolving message bool Override file-level evolution setting
(fory).use_record_for_java message bool Generate Java record for this message
(fory).deprecated message, enum bool Mark type as deprecated
(fory).namespace message string Override default package-based namespace

Field-Level Options

Option Type Description
(fory).ref bool Enable reference tracking for this field
(fory).nullable bool Treat field as nullable (optional)
(fory).weak_ref bool Generate weak pointer semantics (C++/Rust codegen)
(fory).thread_safe_pointer bool Rust ref carrier selection; default true uses Arc/ArcWeak, explicit false uses Rc/RcWeak
(fory).deprecated bool Mark field as deprecated
(fory).type string Primitive override for tagged 64-bit integer encoding

Reference option behavior:

  • weak_ref = true implies ref tracking.
  • For repeated fields, (fory).ref = true applies to list elements.
  • For map<K, V> fields, (fory).ref = true applies to map values.
  • weak_ref and thread_safe_pointer are codegen hints for C++/Rust.
  • thread_safe_pointer defaults to true; it changes only the generated Rust pointer carrier and does not change the wire format.
  • In Rust codegen, (fory).weak_ref = true uses ArcWeak by default and switches to RcWeak only when (fory).thread_safe_pointer = false.

Option Examples by Shape

message Graph {
  Node root = 1 [(fory).ref = true];
  repeated Node nodes = 2 [(fory).ref = true];
  map<string, Node> cache = 3 [(fory).ref = true];
  Node parent = 4 [(fory).weak_ref = true];
  Node local = 5 [(fory).ref = true, (fory).thread_safe_pointer = false];
}

Reference Tracking vs Protobuf IDs

Protobuf itself does not preserve shared/cyclic object graphs. With Fory protobuf extensions, you can opt into graph semantics.

Without Fory ref options (protobuf-style IDs):

message TreeNode {
  string id = 1;
  string parent_id = 2;
  repeated string child_ids = 3;
}

With Fory ref options (object graph):

message TreeNode {
  TreeNode parent = 1 [(fory).weak_ref = true];
  repeated TreeNode children = 2 [(fory).ref = true];
}

Porting Protobuf Schemas To Fory

Step 1: Translate Schema Syntax

  • Keep package names stable.
  • Replace repeated T with list<T> (or keep repeated alias).
  • Add explicit [id=...] where you need stable numeric registration.

Step 2: Convert oneof and Special Types

  • oneof -> union + optional union field.
  • Map protobuf well-known types to Fory primitives (timestamp, duration, any).

Step 3: Replace Protobuf Workarounds with ref

Where protobuf used manual ID links for object graphs, switch to Fory ref modifiers (and optional ref(weak=true) where needed).

Step 4: Update Build/Codegen

Replace protobuf generation steps with the Fory compiler invocation for target languages.

For supported service outputs, add --grpc to emit gRPC companion code:

foryc api.proto --java_out=./generated/java --python_out=./generated/python --go_out=./generated/go --rust_out=./generated/rust --csharp_out=./generated/csharp --dart_out=./generated/dart --scala_out=./generated/scala --kotlin_out=./generated/kotlin --javascript_out=./generated/javascript --grpc

Generated Java service files compile against grpc-java, generated Python service modules use grpc.aio by default, generated Rust service files import tonic and bytes, generated Go service files import grpc-go, generated JavaScript Node.js service files import @grpc/grpc-js, generated C# service files import Grpc.Core.Api types, generated Dart service files import package:grpc, generated Scala service files compile against grpc-java, and generated Kotlin service files compile against grpc-java and grpc-kotlin. Add those dependencies in your application build; Fory packages do not add gRPC as a hard dependency. Use --grpc-python-mode=sync for sync Python grpcio companions. Use --grpc-web with JavaScript output to generate browser clients that import grpc-web. Protobuf oneof fields are translated to Fory union fields inside request and response messages. Direct union RPC request or response types are not part of normal protobuf RPC syntax.

Step 5: Run Compatibility Checks

For staged transitions, keep both formats in parallel and verify payload-level parity with integration tests.

Coexistence Strategy

You can run protobuf and Fory in parallel during a staged transition:

public byte[] serialize(Object obj, Format format) {
    if (format == Format.PROTOBUF) {
        return ((MessageLite) obj).toByteArray();
    }
    return fory.serialize(obj);
}

Use translators at service boundaries while internal object-graph heavy paths migrate first.

Performance References

Summary

Use protobuf when your primary concern is API contracts and gRPC ecosystem integration. Use Fory when object-graph performance, native models, and reference semantics are the primary concern.