OCaml Marshal Codec for MoonBit

A MoonBit implementation for decoding and encoding OCaml's Marshal binary format, enabling interoperability between OCaml and MoonBit programs.

Overview

This library provides a decoder and a faithful encoder for OCaml's Marshal format, OCaml's native binary serialization format. It can decode most common OCaml data types including integers, strings, floats, arrays, tuples, records, and shared data references, and re-encode them.

The encoder (marshal) is byte-identical to OCaml 4.14.1 Marshal.to_string v [] for the supported types: it reproduces OCaml's smallest-fit encodings, the shared-object table with backward-distance references (so DAGs and cycles round-trip), the heap-word size accounting, and the 20-byte header. Round-trip marshal(decode(x)) == x holds on genuine OCaml fixtures.

Features

✅ Supported Data Types

Integers: All ranges from small integers (0-63) to 64-bit integers
Strings: All sizes from small strings (<32 chars) to large strings
Floats: Single doubles and float arrays
Blocks: Tuples, records, variants, lists (tag-based structures)
Float Arrays: Native float arrays with proper endianness handling
Shared References: Handles OCaml's object sharing mechanism
Custom Blocks: Int32, Int64, Nativeint (with both fixed and length-prefixed formats)

⚠️ Not Yet Supported

Custom blocks with custom serializers (e.g., Bigarray)
Code pointers and closures
Big header format (for objects >4GB)

Installation

Add this package to your MoonBit project:

moon add bobzhang/unmarshal

Packages

The module is split so you can depend on only what you need:

Package	Provides
`bobzhang/unmarshal`	shared value model: `MarshalValue`, `MarshalHeader`
`bobzhang/unmarshal/decoder`	`Decoder` (decode marshal bytes)
`bobzhang/unmarshal/encoder`	`marshal` (encode to marshal bytes)
`bobzhang/unmarshal/viz`	`marshal_to_dot` (Graphviz visualization)
`bobzhang/unmarshal/dot`	small Graphviz/Mermaid graph builder

Imported as @unmarshal, @decoder, @encoder, @viz respectively.

Usage

Encoding (Marshaling)

marshal(value) serializes a MarshalValue back to OCaml Marshal bytes, byte-identical to OCaml's own output for the supported types:

///|
test "marshal_round_trip" {
  // Build a value and serialize it (the bytes match `Marshal.to_string`).
  let value = @unmarshal.MBlock(tag=0, [
    @unmarshal.MInt(42),
    @unmarshal.MString(b"hi"),
  ])
  let bytes = @encoder.marshal(value)

  // Decoding the bytes gives the value back.
  let (_, decoded) = @decoder.Decoder::new(bytes).decode()
  inspect(
    decoded,
    content=(
      #|MBlock(tag=0, [MInt(42), MString(<Bytes: [0x68, 0x69]>)])
    ),
  )
}

Physically shared sub-values (and cycles) are encoded as backward references, exactly as OCaml does — share a MarshalValue between fields and the encoder emits a single copy plus shared references.

Basic Example

///|
test "basic_usage" {
  let data : Bytes = [
    b'\x84', b'\x95', b'\xa6', b'\xbe', // Magic number
     b'\x00', b'\x00', b'\x00', b'\x01', // Data length: 1
     b'\x00', b'\x00', b'\x00', b'\x00', // Num objects: 0
     b'\x00', b'\x00', b'\x00', b'\x00', // Size 32: 0
     b'\x00', b'\x00', b'\x00', b'\x00', // Size 64: 0
     b'\x41', // Data: small int 1
  ]
  let decoder = @decoder.Decoder::new(data)
  let (header, value) = decoder.decode()

  // Verify the result
  inspect(header.magic, content="2224400062")
  inspect(value, content="MInt(1)")
}

Decoding Different Data Types

Integers

///|
test "decode_integers" {
  // Small integer (0-63): single byte encoding
  let small_int_data : Bytes = [
    b'\x84', b'\x95', b'\xa6', b'\xbe', b'\x00', b'\x00', b'\x00', b'\x01', b'\x00',
    b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x00',
    b'\x00', b'\x00', b'\x6a', // 0x40 + 42 = 0x6a
  ]
  let decoder = @decoder.Decoder::new(small_int_data)
  let (_, value) = decoder.decode()
  inspect(value, content="MInt(42)")
}

Strings

///|
test "decode_strings" {
  // Small string "Hello" (< 32 chars)
  let string_data : Bytes = [
    b'\x84', b'\x95', b'\xa6', b'\xbe', b'\x00', b'\x00', b'\x00', b'\x06', b'\x00',
    b'\x00', b'\x00', b'\x01', b'\x00', b'\x00', b'\x00', b'\x03', b'\x00', b'\x00',
    b'\x00', b'\x02', b'\x25', // PREFIX_SMALL_STRING + 5
     b'\x48', b'\x65', b'\x6c', b'\x6c', b'\x6f', // "Hello"
  ]
  let decoder = @decoder.Decoder::new(string_data)
  let (_, value) = decoder.decode()
  inspect(
    value,
    content=(
      #|MString(b"Hello")
    ),
  )
}

Tuples and Blocks

///|
test "decode_tuple" {
  // Tuple (1, 2)
  let tuple_data : Bytes = [
    b'\x84', b'\x95', b'\xa6', b'\xbe', b'\x00', b'\x00', b'\x00', b'\x03', b'\x00',
    b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x00',
    b'\x00', b'\x00', b'\xa0', // PREFIX_SMALL_BLOCK: tag=0, size=2
     b'\x41', // Small int 1
     b'\x42', // Small int 2
  ]
  let decoder = @decoder.Decoder::new(tuple_data)
  let (_, value) = decoder.decode()
  match value {
    @unmarshal.MBlock(tag~, fields) => {
      inspect(tag, content="0")
      inspect(fields.length(), content="2")
      inspect(fields[0], content="MInt(1)")
      inspect(fields[1], content="MInt(2)")
    }
    _ => abort("Expected MBlock")
  }
}

Float Arrays

///|
test "decode_float_array" {
  // Simple float array [3.14, 2.71]
  let float_array_data : Bytes = [
    b'\x84', b'\x95', b'\xa6', b'\xbe', b'\x00', b'\x00', b'\x00', b'\x12', b'\x00',
    b'\x00', b'\x00', b'\x01', b'\x00', b'\x00', b'\x00', b'\x05', b'\x00', b'\x00',
    b'\x00', b'\x03', b'\x0e', b'\x02', // CODE_DOUBLE_ARRAY8_LITTLE, count=2
    // First double: 3.14 (little-endian)
     b'\x1f', b'\x85', b'\xeb', b'\x51', b'\xb8', b'\x1e', b'\x09', b'\x40',
    // Second double: 2.71 (little-endian)
     b'\x29', b'\x5c', b'\x8f', b'\xc2', b'\xf5', b'\xa8', b'\x05', b'\x40',
  ]
  let decoder = @decoder.Decoder::new(float_array_data)
  let (_, value) = decoder.decode()
  match value {
    @unmarshal.MDoubleArray(arr) => {
      inspect(arr.length(), content="2")
      // Values are approximately 3.14 and 2.71
      assert_true(arr[0] > 3.13 && arr[0] < 3.15)
      assert_true(arr[1] > 2.70 && arr[1] < 2.72)
    }
    _ => abort("Expected MDoubleArray")
  }
}

Custom Blocks (Int32, Int64, Nativeint)

OCaml's custom blocks allow specialized types like Int32, Int64, and Nativeint to be marshaled:

///|
test "decode_int32" {
  // OCaml Int32.of_int 42
  let int32_data : Bytes = [
    b'\x84', b'\x95', b'\xa6', b'\xbe', b'\x00', b'\x00', b'\x00', b'\x08', b'\x00',
    b'\x00', b'\x00', b'\x01', b'\x00', b'\x00', b'\x00', b'\x03', b'\x00', b'\x00',
    b'\x00', b'\x03', b'\x19', // CODE_CUSTOM_FIXED
     b'\x5f', b'\x69', b'\x00', // "_i" identifier (null-terminated)
     b'\x00', b'\x00', b'\x00', b'\x2a', // 42 in big-endian
  ]
  let decoder = @decoder.Decoder::new(int32_data)
  let (_, value) = decoder.decode()
  match value {
    @unmarshal.MCustom(id, data) => {
      inspect(id, content="_i")
      // Extract the Int32 value (big-endian)
      let val = (data[0].to_int() << 24) |
        (data[1].to_int() << 16) |
        (data[2].to_int() << 8) |
        data[3].to_int()
      inspect(val, content="42")
    }
    _ => abort("Expected MCustom")
  }
}

///|
test "decode_int64" {
  // OCaml Int64.of_int 1000000
  let int64_data : Bytes = [
    b'\x84', b'\x95', b'\xa6', b'\xbe', b'\x00', b'\x00', b'\x00', b'\x0c', b'\x00',
    b'\x00', b'\x00', b'\x01', b'\x00', b'\x00', b'\x00', b'\x03', b'\x00', b'\x00',
    b'\x00', b'\x02', b'\x19', // CODE_CUSTOM_FIXED
     b'\x5f', b'\x6a', b'\x00', // "_j" identifier (null-terminated)
     b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x0f', b'\x42', b'\x40', // 1000000
  ]
  let decoder = @decoder.Decoder::new(int64_data)
  let (_, value) = decoder.decode()
  match value {
    @unmarshal.MCustom(id, [i64be(val)]) => {
      inspect(id, content="_j")
      inspect(val, content="1000000")
    }
    _ => abort("Expected MCustom")
  }
}

///|
test "decode_nativeint" {
  // OCaml Nativeint.of_int 42 (on 64-bit platform)
  let nativeint_data : Bytes = [
    b'\x84', b'\x95', b'\xa6', b'\xbe', b'\x00', b'\x00', b'\x00', b'\x0c', b'\x00',
    b'\x00', b'\x00', b'\x01', b'\x00', b'\x00', b'\x00', b'\x03', b'\x00', b'\x00',
    b'\x00', b'\x02', b'\x19', // CODE_CUSTOM_FIXED
     b'\x5f', b'\x6e', b'\x00', // "_n" identifier (null-terminated)
     b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x00', b'\x2a', // 42
  ]
  let decoder = @decoder.Decoder::new(nativeint_data)
  let (_, value) = decoder.decode()
  match value {
    @unmarshal.MCustom(id, [i64be(val)]) => {
      inspect(id, content="_n")
      inspect(val, content="42")
    }
    _ => abort("Expected MCustom")
  }
}

Custom block identifiers:

"_i" - Int32
"_j" - Int64
"_n" - Nativeint

Working with Shared References

OCaml's Marshal format supports object sharing to avoid duplicating data and handle cyclic structures:

///|
test "shared_references" {
  // Tuple with shared string: ("shared", "shared")
  // The second string is a reference to the first
  let shared_data : Bytes = [
    b'\x84', b'\x95', b'\xa6', b'\xbe', b'\x00', b'\x00', b'\x00', b'\x0a', b'\x00',
    b'\x00', b'\x00', b'\x02', b'\x00', b'\x00', b'\x00', b'\x06', b'\x00', b'\x00',
    b'\x00', b'\x05', b'\xa0', // Small block, tag=0, size=2
     b'\x26', b'\x73', b'\x68', b'\x61', // Small string "shar"
     b'\x72', b'\x65', b'\x64', // "ed"
     b'\x04', b'\x01', // SHARED8, index 1
  ]
  let decoder = @decoder.Decoder::new(shared_data)
  let (_, value) = decoder.decode()
  match value {
    @unmarshal.MBlock(tag~, fields) => {
      inspect(tag, content="0")
      inspect(
        fields[0],
        content=(
          #|MString(b"shared")
        ),
      )
      inspect(
        fields[1],
        content=(
          #|MString(b"shared")
        ),
      ) // Reference to first field
    }
    _ => abort("Expected MBlock")
  }
}

API Reference

Types

`MarshalHeader`

Contains metadata about the marshaled data:

magic : UInt - Magic number (0x8495A6BE or 0x8495A6BF)
data_len : Int - Length of data section
num_objects : Int - Number of shared objects
size_32 : Int - Size on 32-bit platforms
size_64 : Int - Size on 64-bit platforms

`MarshalValue`

Represents decoded OCaml values:

MInt(Int) - Integer values
MString(Bytes) - String/bytes
MFloat(Double) - Floating point
MDoubleArray(Array[Double]) - Float array
MBlock(tag~ : Int, Array[MarshalValue]) - Structured data (tuples, records, variants)
MCustom(String, Bytes) - Custom blocks (Int32, Int64, Nativeint, etc.)

Decoder Functions

`Decoder::new(data : Bytes) -> Decoder`

Creates a new decoder from marshal data.

`Decoder::decode(self : Decoder) -> (MarshalHeader, MarshalValue) raise`

Decodes the marshal data, returning the header and value. Raises an error if the data is malformed.

Encoder Functions

`marshal(value : MarshalValue) -> Bytes raise`

Serializes a MarshalValue to OCaml Marshal wire format (small header), byte-identical to Marshal.to_string value [] for the supported types. Physical sharing between sub-values is reproduced as backward references, so DAGs and cycles round-trip. Raises on an unsupported custom block identifier (only the _i/_j/_n scalars are supported).

Implementation Details

Encoding Format

The OCaml Marshal format uses a tag-based encoding system:

Small integers (0-63): Single byte 0x40 + n
Small strings (<32 chars): 0x20 + len followed by data
Small blocks: 0x80 + tag + (size << 4) for tag < 16, size < 8
Larger values: Use specific code tags (INT8, INT16, STRING8, etc.)

Endianness

Integers are stored in big-endian format
Floats can be either big-endian or little-endian (indicated by different tags)
The decoder handles both endianness variants automatically

Shared Objects

Every decoded object (except shared references themselves) is registered in an internal object table. When a shared reference is encountered, it points to an index in this table, enabling:

Memory-efficient representation of repeated values
Support for cyclic data structures
Preservation of object identity

Testing

Run the test suite:

moon test

Generate test data from OCaml:

ocaml test_generator.ml > marshal_data_test.mbt

Contributing

Contributions are welcome! Areas that need work:

Custom blocks with custom serializers (Bigarray, etc.)
Big header format for large objects
Better error messages with position information
More comprehensive test coverage
Performance optimizations

License

This project is licensed under the Apache-2.0 License - see the LICENSE file for details.

Acknowledgments

This implementation is based on the OCaml Marshal format specification and the OCaml runtime's extern.c implementation.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
cmd/main		cmd/main
decoder		decoder
dot		dot
encoder		encoder
viz		viz
.gitignore		.gitignore
Agents.md		Agents.md
LICENSE		LICENSE
README.mbt.md		README.mbt.md
README.md		README.md
TODO_MISSING_FORMATS.md		TODO_MISSING_FORMATS.md
Task.md		Task.md
data_generate.ml		data_generate.ml
generate_custom_blocks.ml		generate_custom_blocks.ml
moon.mod		moon.mod
moon.pkg		moon.pkg
ocaml_marshalling_format.md		ocaml_marshalling_format.md
pkg.generated.mbti		pkg.generated.mbti
test_generator.ml		test_generator.ml
types.mbt		types.mbt
unmarshal.mbt		unmarshal.mbt

Latest commit

History

Folders and files

Repository files navigation

OCaml Marshal Codec for MoonBit

Overview

Features

✅ Supported Data Types

⚠️ Not Yet Supported

Installation

Packages

Usage

Encoding (Marshaling)

Basic Example

Decoding Different Data Types

Integers

Strings

Tuples and Blocks

Float Arrays

Custom Blocks (Int32, Int64, Nativeint)

Working with Shared References

API Reference

Types

MarshalHeader

MarshalValue

Decoder Functions

Decoder::new(data : Bytes) -> Decoder

Decoder::decode(self : Decoder) -> (MarshalHeader, MarshalValue) raise

Encoder Functions

marshal(value : MarshalValue) -> Bytes raise

Implementation Details

Encoding Format

Endianness

Shared Objects

Testing

Contributing

License

Acknowledgments

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`MarshalHeader`

`MarshalValue`

`Decoder::new(data : Bytes) -> Decoder`

`Decoder::decode(self : Decoder) -> (MarshalHeader, MarshalValue) raise`

`marshal(value : MarshalValue) -> Bytes raise`