Repository navigation
Conversation
An NDArrayWrapper of strings failed with `KeyError: key String not found`.
Convert it to UCS4String{N}, N the longest string's length in characters
(at least 1, like NumPy), so it is written as a `ucs4` ndarray the way
Python asdf writes a unicode array. Inline storage now accepts strings,
and invalid characters throw an ArgumentError.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #88 +/- ##
==========================================
+ Coverage 99.30% 99.31% +0.01%
==========================================
Files 1 1
Lines 717 729 +12
==========================================
+ Hits 712 724 +12
Misses 5 5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thanks Chris! Here are some review comments with individual patches that you can edit and apply (in order) as you see fit. I've also included a diff at the bottom with everything combined.
The snippets assume wl(x; kw...) = (ASDF.write_file(p, OrderedDict("a" => ASDF.NDArrayWrapper(x; kw...))); ASDF.load_file(p)["a"][])1. Inline strings don't round-tripProblem. YAML.jl's ASDF.write_file(p, OrderedDict("a" => ASDF.NDArrayWrapper(["\"\0x", "ok"]; inline = true), "b" => ASDF.NDArrayWrapper([1, 2]), "after" => "kept"))
collect(keys(ASDF.load_file(p).metadata)) # Any["a"]: "b", "after" and asdf_library are gone
ASDF.load_file(p)["a"][] # ["\""]
All of these round-trip as blocks. Fix. String elements now go through data:
- "\"\u0000x"
- "co\u00adop"
- "don\u0092t"
- "\u0085"PyYAML 6.0.1 reads all 13 test strings back unchanged. I couldn't check Python The fix also removes a slowdown: YAML.jl's string printer was compiled once per The round-trip test, run both as blocks and inline, gets an Patch 1 of 12diff --git a/src/ASDF.jl b/src/ASDF.jl
index d5631d8..d59040d 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -1366,10 +1366,33 @@ struct YAMLScalar
end
YAML._print(io::IO, val::YAMLScalar, level::Int = 0, ignore_level::Bool = false) = println(io, val.text)
-# Rewrite inline ndarray data so every float is emitted in YAML-1.1-compliant form. Document
-# metadata gets the same treatment from `_convert_tree`. The data nests one list per dimension, the
-# first `shape` (last Julia) dimension outermost.
+# A string as a YAML double-quoted scalar with every unprintable character, line breaks included,
+# escaped at full width. YAML.jl's own string printer cannot round-trip arbitrary strings: Julia's
+# short escapes (`\u85`) are invalid YAML, and its block literals drop leading spaces and CRs and
+# write NUL raw, which ends the YAML stream.
+function yaml_quoted_string(s::String)
+ return sprint() do io
+ print(io, '"')
+ for c in s
+ if c == '"' || c == '\\'
+ print(io, '\\', c)
+ elseif isprint(c)
+ print(io, c)
+ elseif c <= '\uffff'
+ print(io, "\\u", string(UInt32(c); base = 16, pad = 4))
+ else
+ print(io, "\\U", string(UInt32(c); base = 16, pad = 8))
+ end
+ end
+ print(io, '"')
+ end
+end
+
+# Rewrite inline ndarray data so every float is emitted in YAML-1.1-compliant form, and every
+# string round-trips. Document metadata floats get the same treatment from `_convert_tree`. The data
+# nests one list per dimension, the first `shape` (last Julia) dimension outermost.
yaml_compliant(val::AbstractFloat) = YAMLScalar(yaml_float_string(val))
+yaml_compliant(val::ASDFString) = YAMLScalar(yaml_quoted_string(String(val)))
yaml_compliant(val::AbstractVector) = map(yaml_compliant, val)
yaml_compliant(val::AbstractArray) = map(yaml_compliant, eachslice(val; dims = ndims(val)))
yaml_compliant(val) = val
@@ -1910,7 +1933,7 @@ function YAML._print(io::IO, val::NDArrayWrapper, level::Int = 0, ignore_level::
shape = collect(Int64, reverse(size(val.array)))
if val.inline
ndarray = OrderedDict(
- # Route inline data through the float pass so its elements get YAML-1.1 exponents too.
+ # Route inline data through `yaml_compliant` so floats get YAML-1.1 exponents and strings round-trip.
# `collect` first: `eachslice` misindexes views of views with `Base.IdentityUnitRange` indices.
:data => yaml_compliant(collect(val.array)),
:shape => shape,
diff --git a/test/test-write.jl b/test/test-write.jl
index facdc0d..392db6b 100644
--- a/test/test-write.jl
+++ b/test/test-write.jl
@@ -205,8 +205,11 @@ end
# Like a NumPy unicode array, a wrapped string array is a `ucs4` ndarray as wide as its longest string,
# and at least 1 wide. A plain vector of strings stays a YAML list.
path = joinpath(mktempdir(; cleanup = true), "strings.asdf")
- arrays = Dict{Any, Any}("matrix" => ["" "right"; "left" ""], "unicode" => ["é", "🎉x", "true", "1", "a\nb", "\"q\""], "empty_strings" => ["", ""], "no_strings" => String[], "zero_dimensional" => fill("abc"), "substrings" => split("a bc"))
- widths = Dict("matrix" => 5, "unicode" => 4, "empty_strings" => 1, "no_strings" => 1, "zero_dimensional" => 3, "substrings" => 2)
+ # `escapes` holds strings that YAML.jl's string printer can't round-trip: NUL ends its YAML stream,
+ # Julia's short `\u` escapes are invalid YAML, and block literals lose leading spaces and CRs.
+ escapes = ["\"\0x", "co\u00adop", "don\u0092t", "\u85", "\U1f3f4\Ue0067\Ue0062\Ue0073\Ue0063\Ue0074\Ue007f", " a\nb", "a\r\nb", " \"x", "x ", "\\n", "\t\u2028\ufeff"]
+ arrays = Dict{Any, Any}("matrix" => ["" "right"; "left" ""], "unicode" => ["é", "🎉x", "true", "1", "a\nb", "\"q\""], "escapes" => escapes, "empty_strings" => ["", ""], "no_strings" => String[], "zero_dimensional" => fill("abc"), "substrings" => split("a bc"))
+ widths = Dict("matrix" => 5, "unicode" => 4, "escapes" => 7, "empty_strings" => 1, "no_strings" => 1, "zero_dimensional" => 3, "substrings" => 2)
for inline in (false, true)
written = inline ? delete!(copy(arrays), "zero_dimensional") : arrays
ASDF.write_file(path, Dict{Any, Any}("list" => ["a", "b"], (key => ASDF.NDArrayWrapper(array; inline) for (key, array) in written)...))2. Inline
|
Problem
NDArrayWrapper(["a" "bc"; "def" ""])fails at write time withKeyError: key String not found, andinline = truerejects string arrays. Pythonasdfwrites a NumPy unicode array as aucs4ndarray, and some schemas require one (GWCS label maps, for example), so these files can't be written from Julia.Changes
UCS4String{N}, whereNis the longest string's length in characters (at least 1, as in NumPy). The existingucs4path then writes it, so blocks, byte order, and compression work as before.inline = trueaccepts string arrays, in the form Pythonasdfwrites.ArgumentError.A plain
Vector{String}is still written as a YAML list, and other element types are written exactly as before.Tests
String matrices with non-ASCII and non-BMP characters, empty strings, empty and 0-dimensional arrays,
SubStrings, block and inline storage, zlib compression, and the invalid-character error. Pythonasdf5.3.1 reads every file these tests write.🤖 Generated with Claude Code