Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@

A new [Advanced Scientific Data Format (ASDF)](https://asdf-standard.readthedocs.io/en/latest/index.html) package, written in Julia.

Packages can extend `ASDF.to_tree` to embed their own Julia objects anywhere in
a larger ASDF document alongside ordinary metadata and binary arrays. See the
[custom type documentation](https://juliaastro.org/ASDF.jl/dev/custom-types/)
for the write-conversion interface.

## Quickstart

```julia
Expand Down
1 change: 1 addition & 0 deletions docs/make.jl
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ makedocs(;
"JWST" => "examples/jwst.md",
"Roman" => "examples/roman.md",
],
"Custom Julia types" => "custom-types.md",
"Interoperability" => "interop.md",
"API" => "api.md",
],
Expand Down
11 changes: 11 additions & 0 deletions docs/src/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,19 @@ Modules = [ASDF]
Private = false
```

## Custom type writing

```@docs
ASDF.WriteContext
ASDF.to_tree
ASDF.TaggedMapping
ASDF.TaggedSequence
ASDF.TaggedScalar
```

## Private
```@autodocs
Modules = [ASDF]
Public = false
Filter = value -> value ∉ (ASDF.WriteContext, ASDF.to_tree, ASDF.TaggedMapping, ASDF.TaggedSequence, ASDF.TaggedScalar)
```
89 changes: 89 additions & 0 deletions docs/src/custom-types.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Custom Julia types

ASDF documents often combine metadata, binary arrays, and objects owned by
domain packages. A package can make its types writable anywhere in an ASDF
document by extending [`ASDF.to_tree`](@ref):

```@example custom_types
using ASDF

struct Measurement
value::Float64
unit::String
end

function ASDF.to_tree(measurement::Measurement, context::ASDF.WriteContext)
properties = Dict("value" => measurement.value, "unit" => measurement.unit)
return ASDF.TaggedMapping("tag:example.org/measurement-1.0.0", properties)
end

document = Dict(
"meta" => Dict("exposure" => Measurement(1200.0, "s")),
"data" => ASDF.NDArrayWrapper(reshape(collect(1.0:12.0), 3, 4)),
)

save("custom-types.asdf", document)
```

`save` and [`ASDF.write_file`](@ref) recursively walk the complete document.
When they encounter a `Measurement`, Julia dispatch selects the method above.
ASDF then recursively converts custom objects contained in the returned node
before writing YAML and binary blocks.

The original document is not modified. The same conversion can be inspected
without writing a file:

```@example custom_types
node = ASDF.to_tree(Measurement(5.0, "m"))
node.tag
```

## Conversion contract

The public interface has two forms:

```julia
ASDF.to_tree(value)
ASDF.to_tree(value, context::ASDF.WriteContext)
```

The one-argument form recursively converts a value and is useful for inspecting
the ASDF representation. The two-argument form is the extension hook. Its
fallback returns the value unchanged.

A package method should return one of:

- a scalar, string, mapping, or array already supported by ASDF.jl;
- [`ASDF.TaggedMapping`](@ref), [`ASDF.TaggedSequence`](@ref), or
[`ASDF.TaggedScalar`](@ref);
- [`ASDF.NDArrayWrapper`](@ref) for explicit inline or binary array storage.

Converter methods are shallow. They may return mappings or sequences containing
other custom objects; ASDF.jl converts those children automatically. Methods
should accept the [`ASDF.WriteContext`](@ref) but treat it as opaque.

ASDF.jl rejects cyclic mappings, arrays, or converter output because ASDF
reference serialization is not yet implemented.

## Optional ASDF support

When ASDF.jl is an optional dependency, define the method in a Julia package
extension that loads only when both packages are present:

```julia
module MyPackageASDFExt

using ASDF
using MyPackage

function ASDF.to_tree(value::MyPackage.CustomType, context::ASDF.WriteContext)
return ASDF.TaggedMapping("tag:example.org/custom-1.0.0", Dict("value" => value.value))
end

end
```

This interface only controls writing. Loading an unknown tag with
`extensions = true` produces an `ASDF.TaggedMapping`, `ASDF.TaggedSequence`, or
`ASDF.TaggedScalar`; reconstructing package-owned objects and validating their
schemas require separate read-side support.
5 changes: 4 additions & 1 deletion docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,4 +125,7 @@ af["data"][] == [1, 2, 3, 4]

## Tagged objects

Come back soon to see how custom Julia objects can be handled in ASDF.jl.
Packages can extend [`ASDF.to_tree`](@ref) to serialize their own Julia types
wherever they occur in a larger ASDF document. See [Custom Julia
types](@ref) for the conversion contract and an example combining a custom
metadata object with a binary array.
87 changes: 86 additions & 1 deletion src/ASDF.jl
Original file line number Diff line number Diff line change
Expand Up @@ -1153,6 +1153,86 @@ function YAML._print(io::IO, val::TaggedScalar, level::Int = 0, ignore_level::Bo
return YAML._print(io, val.value, level, ignore_level)
end

"""
WriteContext

Opaque context for converting Julia objects to ASDF-compatible tree nodes.

Packages that extend [`ASDF.to_tree`](@ref) should accept this context but must
not depend on its fields. Future versions may use it for schema selection,
array-storage policies, and extension provenance.
"""
struct WriteContext
_active::IdDict{Any, Nothing}
end
WriteContext() = WriteContext(IdDict{Any, Nothing}())

"""
ASDF.to_tree(value)
ASDF.to_tree(value, context::ASDF.WriteContext)

Convert Julia objects to an ASDF-compatible tree.

The one-argument form recursively converts `value` and any custom objects
nested in mappings, arrays, or tagged nodes. Packages add support for their own
types by defining a two-argument method that returns a scalar, mapping,
sequence, [`ASDF.TaggedMapping`](@ref), [`ASDF.TaggedSequence`](@ref),
[`ASDF.TaggedScalar`](@ref), [`ASDF.NDArrayWrapper`](@ref), or another value
already supported by the ASDF writer. Returned mappings and sequences may
contain additional custom objects; ASDF converts those children recursively.

The fallback two-argument method returns `value` unchanged.
"""
to_tree(value, ::WriteContext) = value
to_tree(value) = _convert_tree(value, WriteContext())

function _with_active(f, value, context::WriteContext)
haskey(context._active, value) && throw(ArgumentError("cyclic ASDF write conversion involving $(typeof(value)) is not supported"))
context._active[value] = nothing
try
return f()
finally
delete!(context._active, value)
end
end

function _convert_tree(value, context::WriteContext)
converted = to_tree(value, context)
converted === value && return _convert_tree_children(converted, context)
return _with_active(value, context) do
_convert_tree_children(converted, context)
end
end

_convert_tree_children(value, context::WriteContext) = value
_convert_tree_children(value::TaggedScalar, context::WriteContext) = value

function _convert_tree_children(value::TaggedMapping, context::WriteContext)
converted = _with_active(value.value, context) do
OrderedDict{Any, Any}(key => _convert_tree(item, context) for (key, item) in value)
end
return TaggedMapping(value.tag, converted)
end

function _convert_tree_children(value::TaggedSequence, context::WriteContext)
converted = _with_active(value.value, context) do
map(item -> _convert_tree(item, context), value.value)
end
return TaggedSequence(value.tag, converted)
end

function _convert_tree_children(value::AbstractDict, context::WriteContext)
return _with_active(value, context) do
OrderedDict{Any, Any}(key => _convert_tree(item, context) for (key, item) in value)
end
end

function _convert_tree_children(value::AbstractArray, context::WriteContext)
return _with_active(value, context) do
map(item -> _convert_tree(item, context), value)
end
end

function YAML._print(io::IO, val::NDArray, level::Int = 0, ignore_level::Bool = false)
# TODO: Get compression from underlying header block?
return YAML._print(io, NDArrayWrapper(val[]; compression = C_None), level, ignore_level)
Expand Down Expand Up @@ -1756,7 +1836,10 @@ end
"""
write_file(filename::AbstractString, document::AbstractDict)

Writes an ASDF file to disk. `document` is a plain `Dict` whose values may include [`NDArrayWrapper`](@ref) instances. These are serialized as binary blocks with appropriate compression.
Writes an ASDF file to disk. `document` may contain custom Julia objects with
two-argument [`ASDF.to_tree`](@ref) methods, including objects nested inside
mappings or arrays. Values may also include [`NDArrayWrapper`](@ref) instances,
which are serialized as binary blocks with appropriate compression.

Layout of the output file:

Expand All @@ -1783,6 +1866,8 @@ function write_file(filename::AbstractString, document::AbstractDict)
# back to an unordered `Dict` and drop the order). The provenance entry is stamped last.
full_document = OrderedDict{Any, Any}(document)
full_document["asdf_library"] = library
# Convert package-owned objects before float normalization and block collection.
full_document = to_tree(full_document)
# Rewrite floats so their exponents are YAML-1.1 compliant (see `yaml_compliant`).
full_document = yaml_compliant(full_document)

Expand Down
120 changes: 120 additions & 0 deletions test/test-write-converters.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
abstract type AbstractWriteValue end

struct WriteValue <: AbstractWriteValue
value::Int
end

struct SpecialWriteValue <: AbstractWriteValue
value::Int
end

struct WriteParent
child
end

struct WriteSequence
values::Vector{Any}
end

struct WriteScalar
value::String
end

struct WriteBlock
data::Matrix{Float64}
end

struct CyclicWriteValue end

ASDF.to_tree(value::AbstractWriteValue, context::ASDF.WriteContext) = ASDF.TaggedMapping("tag:example.org/write/value-1.0.0", OrderedDict("value" => value.value))
ASDF.to_tree(value::SpecialWriteValue, context::ASDF.WriteContext) = ASDF.TaggedMapping("tag:example.org/write/special-1.0.0", OrderedDict("value" => value.value))
ASDF.to_tree(value::WriteParent, context::ASDF.WriteContext) = ASDF.TaggedMapping("tag:example.org/write/parent-1.0.0", OrderedDict("child" => value.child))
ASDF.to_tree(value::WriteSequence, context::ASDF.WriteContext) = ASDF.TaggedSequence("tag:example.org/write/sequence-1.0.0", value.values)
ASDF.to_tree(value::WriteScalar, context::ASDF.WriteContext) = ASDF.TaggedScalar("tag:example.org/write/scalar-1.0.0", value.value)
ASDF.to_tree(value::WriteBlock, context::ASDF.WriteContext) = ASDF.TaggedMapping("tag:example.org/write/block-1.0.0", OrderedDict("data" => ASDF.NDArrayWrapper(value.data; compression = ASDF.C_Zlib)))
ASDF.to_tree(value::CyclicWriteValue, context::ASDF.WriteContext) = OrderedDict("self" => value)

@testset "write conversion protocol" begin
shallow = ASDF.to_tree(WriteParent(WriteValue(3)), ASDF.WriteContext())
@test shallow isa ASDF.TaggedMapping
@test shallow["child"] isa WriteValue

parent = ASDF.to_tree(WriteParent(WriteValue(3)))
@test parent.tag == "tag:example.org/write/parent-1.0.0"
@test parent["child"] isa ASDF.TaggedMapping
@test parent["child"].tag == "tag:example.org/write/value-1.0.0"
@test parent["child"]["value"] == 3

sequence = ASDF.to_tree(WriteSequence(Any[WriteValue(4), "plain"]))
@test sequence isa ASDF.TaggedSequence
@test sequence.tag == "tag:example.org/write/sequence-1.0.0"
@test sequence[1] isa ASDF.TaggedMapping
@test sequence[2] == "plain"

scalar = ASDF.to_tree(WriteScalar("converted"))
@test scalar isa ASDF.TaggedScalar
@test scalar.tag == "tag:example.org/write/scalar-1.0.0"
@test String(scalar) == "converted"

@test ASDF.to_tree(WriteValue(5)).tag == "tag:example.org/write/value-1.0.0"
@test ASDF.to_tree(SpecialWriteValue(5)).tag == "tag:example.org/write/special-1.0.0"
end

@testset "recursive write conversion" begin
tagged = ASDF.TaggedMapping("tag:example.org/write/existing-1.0.0", OrderedDict("child" => WriteValue(6)))
converted = ASDF.to_tree(tagged)
@test converted.tag == tagged.tag
@test converted["child"] isa ASDF.TaggedMapping
@test tagged["child"] isa WriteValue

source = OrderedDict("zebra" => WriteValue(7), "apple" => Any[1, WriteValue(8)], "matrix" => [1 2; 3 4])
plain = ASDF.to_tree(source)
@test collect(keys(plain)) == collect(keys(source))
@test plain["zebra"] isa ASDF.TaggedMapping
@test plain["apple"][2] isa ASDF.TaggedMapping
@test plain["matrix"] == source["matrix"]
@test source["zebra"] isa WriteValue
@test source["apple"][2] isa WriteValue

cyclic_mapping = OrderedDict{Any, Any}()
cyclic_mapping["self"] = cyclic_mapping
@test_throws "cyclic ASDF write conversion" ASDF.to_tree(cyclic_mapping)

cyclic_sequence = Any[]
push!(cyclic_sequence, cyclic_sequence)
@test_throws "cyclic ASDF write conversion" ASDF.to_tree(cyclic_sequence)
@test_throws "cyclic ASDF write conversion" ASDF.to_tree(CyclicWriteValue())
end

@testset "heterogeneous document writing" begin
data = reshape(collect(1.0:12.0), 3, 4)
document = OrderedDict(
"roman" => OrderedDict(
"meta" => OrderedDict("model" => WriteParent(SpecialWriteValue(9)), "description" => "nested"),
"data" => WriteBlock(data),
),
"name" => "product",
)

mktempdir() do directory
filename = joinpath(directory, "custom.asdf")
ASDF.write_file(filename, document)
loaded = ASDF.load_file(filename; extensions = true)

@test collect(keys(loaded.metadata)) == ["roman", "name", "asdf_library"]
@test loaded["name"] == "product"
@test loaded["roman"]["meta"]["model"].tag == "tag:example.org/write/parent-1.0.0"
@test loaded["roman"]["meta"]["model"]["child"].tag == "tag:example.org/write/special-1.0.0"
@test loaded["roman"]["data"].tag == "tag:example.org/write/block-1.0.0"
@test loaded["roman"]["data"]["data"][] == data
@test !haskey(document, "asdf_library")
@test document["roman"]["meta"]["model"] isa WriteParent
@test document["roman"]["data"] isa WriteBlock

save_filename = joinpath(directory, "custom-save.asdf")
save(save_filename, document)
saved = ASDF.load_file(save_filename; extensions = true)
@test saved["roman"]["data"]["data"][] == data
@test saved["roman"]["meta"]["model"]["child"]["value"] == 9
end
end
Loading