Skip to content

Write arrays of strings as ucs4 ndarrays - #88

Open
cgarling wants to merge 1 commit into
mainfrom
string-arrays
Open

cgarling wants to merge 1 commit into
mainfrom
string-arrays

Conversation

@cgarling

Copy link
Copy Markdown
Member

Problem

NDArrayWrapper(["a" "bc"; "def" ""]) fails at write time with KeyError: key String not found, and inline = true rejects string arrays. Python asdf writes a NumPy unicode array as a ucs4 ndarray, and some schemas require one (GWCS label maps, for example), so these files can't be written from Julia.

Changes

  • A wrapped array of strings is converted to UCS4String{N}, where N is the longest string's length in characters (at least 1, as in NumPy). The existing ucs4 path then writes it, so blocks, byte order, and compression work as before.
  • inline = true accepts string arrays, in the form Python asdf writes.
  • A string with an invalid character throws an ArgumentError.

A plain Vector{String} is still written as a YAML list, and other element types are written exactly as before.

Tests

String matrices with non-ASCII and non-BMP characters, empty strings, empty and 0-dimensional arrays, SubStrings, block and inline storage, zlib compression, and the invalid-character error. Python asdf 5.3.1 reads every file these tests write.

🤖 Generated with Claude Code

An NDArrayWrapper of strings failed with `KeyError: key String not found`.
Convert it to UCS4String{N}, N the longest string's length in characters
(at least 1, like NumPy), so it is written as a `ucs4` ndarray the way
Python asdf writes a unicode array. Inline storage now accepts strings,
and invalid characters throw an ArgumentError.
@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.31%. Comparing base (69aa0d3) to head (51bc25f).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #88      +/-   ##
==========================================
+ Coverage   99.30%   99.31%   +0.01%     
==========================================
  Files           1        1              
  Lines         717      729      +12     
==========================================
+ Hits          712      724      +12     
  Misses          5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@icweaver

Copy link
Copy Markdown
Member

Thanks Chris! Here are some review comments with individual patches that you can edit and apply (in order) as you see fit. I've also included a diff at the bottom with everything combined.

# Problem on 51bc25f Fix
1 Some inline strings write, then fail to load or load changed. A NUL drops the rest of the file Write inline strings as double-quoted YAML with full-width escapes
2 Inline ascii arrays with bytes above 0x7f write, then fail to load Reject them before the file is opened
3 Mixed-width string views throw KeyError Convert them to ucs4
4 TaggedScalar arrays lose their tags Reject them
5 A trailing \0 is dropped on load Reject it
6 A string of ~2.1M characters overflows the stack Reject strings longer than max_string_width = 2^16
7 The writer and the inline reader check characters differently Share string_code and string_width
8 The first inline load compiles a new type for each string length Pad with the code type
9 ucs4_array builds each string separately Fill one UInt32 buffer
10 Julia 1.10 writes wide string blocks byte by byte Copy the code units out first
11 The inline test pass never checks that the arrays are inline Assert it, and check that rejected writes leave the file unchanged
12 Docs and comments describe the old behaviour Update them

The snippets assume using ASDF, OrderedCollections, p = tempname(), and this write-then-load helper:

wl(x; kw...) = (ASDF.write_file(p, OrderedDict("a" => ASDF.NDArrayWrapper(x; kw...))); ASDF.load_file(p)["a"][])
1. Inline strings don't round-trip

Problem. YAML.jl's _print(::AbstractString) writes Julia's short escapes (\u85), which YAML rejects. For a string containing " or a newline it switches to a |- block literal, which drops leading spaces and CRs and writes NUL raw. A raw NUL ends the YAML stream:

ASDF.write_file(p, OrderedDict("a" => ASDF.NDArrayWrapper(["\"\0x", "ok"]; inline = true), "b" => ASDF.NDArrayWrapper([1, 2]), "after" => "kept"))
collect(keys(ASDF.load_file(p).metadata))   # Any["a"]: "b", "after" and asdf_library are gone
ASDF.load_file(p)["a"][]                    # ["\""]
wl(x; inline = true) Result on 51bc25f
["co\u00adop"], ["\u85"] YAML.ScannerError
[" a\nb"] YAML.ParserError
["a\r\nb"] ["a\nb"]
[" \"x"] ["\"x"]

All of these round-trip as blocks.

Fix. String elements now go through yaml_compliant, as floats already do. Each one is written as a double-quoted scalar with every unprintable character escaped at full width:

data:
  - "\"\u0000x"
  - "co\u00adop"
  - "don\u0092t"
  - "\u0085"

PyYAML 6.0.1 reads all 13 test strings back unchanged. I couldn't check Python asdf because it isn't installed here.

The fix also removes a slowdown: YAML.jl's string printer was compiled once per UCS4String{N}, so writing 20 inline arrays of distinct widths took 15.9 s. It takes 1.5 s now that the elements reach YAML.jl as YAMLScalars.

The round-trip test, run both as blocks and inline, gets an escapes array of such strings.

Patch 1 of 12
diff --git a/src/ASDF.jl b/src/ASDF.jl
index d5631d8..d59040d 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -1366,10 +1366,33 @@ struct YAMLScalar
 end
 YAML._print(io::IO, val::YAMLScalar, level::Int = 0, ignore_level::Bool = false) = println(io, val.text)
 
-# Rewrite inline ndarray data so every float is emitted in YAML-1.1-compliant form. Document
-# metadata gets the same treatment from `_convert_tree`. The data nests one list per dimension, the
-# first `shape` (last Julia) dimension outermost.
+# A string as a YAML double-quoted scalar with every unprintable character, line breaks included,
+# escaped at full width. YAML.jl's own string printer cannot round-trip arbitrary strings: Julia's
+# short escapes (`\u85`) are invalid YAML, and its block literals drop leading spaces and CRs and
+# write NUL raw, which ends the YAML stream.
+function yaml_quoted_string(s::String)
+    return sprint() do io
+        print(io, '"')
+        for c in s
+            if c == '"' || c == '\\'
+                print(io, '\\', c)
+            elseif isprint(c)
+                print(io, c)
+            elseif c <= '\uffff'
+                print(io, "\\u", string(UInt32(c); base = 16, pad = 4))
+            else
+                print(io, "\\U", string(UInt32(c); base = 16, pad = 8))
+            end
+        end
+        print(io, '"')
+    end
+end
+
+# Rewrite inline ndarray data so every float is emitted in YAML-1.1-compliant form, and every
+# string round-trips. Document metadata floats get the same treatment from `_convert_tree`. The data
+# nests one list per dimension, the first `shape` (last Julia) dimension outermost.
 yaml_compliant(val::AbstractFloat) = YAMLScalar(yaml_float_string(val))
+yaml_compliant(val::ASDFString) = YAMLScalar(yaml_quoted_string(String(val)))
 yaml_compliant(val::AbstractVector) = map(yaml_compliant, val)
 yaml_compliant(val::AbstractArray) = map(yaml_compliant, eachslice(val; dims = ndims(val)))
 yaml_compliant(val) = val
@@ -1910,7 +1933,7 @@ function YAML._print(io::IO, val::NDArrayWrapper, level::Int = 0, ignore_level::
     shape = collect(Int64, reverse(size(val.array)))
     if val.inline
         ndarray = OrderedDict(
-            # Route inline data through the float pass so its elements get YAML-1.1 exponents too.
+            # Route inline data through `yaml_compliant` so floats get YAML-1.1 exponents and strings round-trip.
             # `collect` first: `eachslice` misindexes views of views with `Base.IdentityUnitRange` indices.
             :data => yaml_compliant(collect(val.array)),
             :shape => shape,
diff --git a/test/test-write.jl b/test/test-write.jl
index facdc0d..392db6b 100644
--- a/test/test-write.jl
+++ b/test/test-write.jl
@@ -205,8 +205,11 @@ end
     # Like a NumPy unicode array, a wrapped string array is a `ucs4` ndarray as wide as its longest string,
     # and at least 1 wide. A plain vector of strings stays a YAML list.
     path = joinpath(mktempdir(; cleanup = true), "strings.asdf")
-    arrays = Dict{Any, Any}("matrix" => ["" "right"; "left" ""], "unicode" => ["é", "🎉x", "true", "1", "a\nb", "\"q\""], "empty_strings" => ["", ""], "no_strings" => String[], "zero_dimensional" => fill("abc"), "substrings" => split("a bc"))
-    widths = Dict("matrix" => 5, "unicode" => 4, "empty_strings" => 1, "no_strings" => 1, "zero_dimensional" => 3, "substrings" => 2)
+    # `escapes` holds strings that YAML.jl's string printer can't round-trip: NUL ends its YAML stream,
+    # Julia's short `\u` escapes are invalid YAML, and block literals lose leading spaces and CRs.
+    escapes = ["\"\0x", "co\u00adop", "don\u0092t", "\u85", "\U1f3f4\Ue0067\Ue0062\Ue0073\Ue0063\Ue0074\Ue007f", "  a\nb", "a\r\nb", " \"x", "x ", "\\n", "\t\u2028\ufeff"]
+    arrays = Dict{Any, Any}("matrix" => ["" "right"; "left" ""], "unicode" => ["é", "🎉x", "true", "1", "a\nb", "\"q\""], "escapes" => escapes, "empty_strings" => ["", ""], "no_strings" => String[], "zero_dimensional" => fill("abc"), "substrings" => split("a bc"))
+    widths = Dict("matrix" => 5, "unicode" => 4, "escapes" => 7, "empty_strings" => 1, "no_strings" => 1, "zero_dimensional" => 3, "substrings" => 2)
     for inline in (false, true)
         written = inline ? delete!(copy(arrays), "zero_dimensional") : arrays
         ASDF.write_file(path, Dict{Any, Any}("list" => ["a", "b"], (key => ASDF.NDArrayWrapper(array; inline) for (key, array) in written)...))
2. Inline ascii arrays with non-ASCII bytes can't be loaded

Problem. The guard let AsciiString arrays through inline without checking for bytes above 0x7f, which the inline reader rejects. A loaded ascii block holding NumPy S data in Latin-1 is the realistic case:

wl([ASDF.AsciiString((0x61, 0xe9))]; inline = true)   # the write replaces the file; the load throws ArgumentError: inline string "aé" is not ASCII

Fix. Reject such arrays before the file is opened. Block storage round-trips them and is unchanged.

A test for the block round trip is in 3.

Patch 2 of 12
diff --git a/src/ASDF.jl b/src/ASDF.jl
index d59040d..98e75a6 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -1858,6 +1858,8 @@ function _convert_tree_children(value::NDArrayWrapper, active)
     # Inline elements are YAML scalars, which YAML.jl reads back only as booleans, `Int64`s, floats, and strings.
     eltype(array) <: Union{Integer, AbstractFloat, ASDFString} || throw(ArgumentError("an array of $(eltype(array)) cannot be stored inline; use `inline = false`"))
     eltype(array) <: UInt64 && any(>(typemax(Int64)), array) && throw(ArgumentError("`UInt64` values above `typemax(Int64)` cannot be stored inline; use `inline = false`"))
+    # The inline reader accepts only ASCII `ascii` strings.
+    eltype(array) <: AsciiString && !all(isascii, array) && throw(ArgumentError("`ascii` strings with non-ASCII bytes cannot be stored inline; use `inline = false`"))
     return value
 end
 
diff --git a/test/test-write.jl b/test/test-write.jl
index 392db6b..5d6d7b1 100644
--- a/test/test-write.jl
+++ b/test/test-write.jl
@@ -195,7 +195,7 @@ end
     path = joinpath(mktempdir(; cleanup = true), "existing.asdf")
     ASDF.write_file(path, Dict{Any, Any}("kept" => 1))
     contents = read(path)
-    for array in (fill(1.5), ComplexF64[1 + 2im], [(0x61, 0x62)], [(a = 1, b = 2.0)], UInt64[typemax(UInt64)])
+    for array in (fill(1.5), ComplexF64[1 + 2im], [(0x61, 0x62)], [(a = 1, b = 2.0)], UInt64[typemax(UInt64)], [ASDF.AsciiString((0x61, 0xe9))])
         @test_throws "cannot be stored inline; use `inline = false`" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(array; inline = true)))
         @test read(path) == contents
     end
3. Mixed-width string views throw KeyError

Problem. !(eltype <: ASDFString) also passed non-concrete string-view eltypes through, and infer_asdf_datatype can't map them. So a vcat of loaded ucs4 arrays of different widths failed, inline or not:

wl(ASDF.UCS4String[ASDF.UCS4String((0x00000061,)), ASDF.UCS4String((0x00000062, 0x00000063))])   # KeyError: key ASDF.UCS4String not found

Fix. Decide by dispatch. Only arrays of a concrete UCS4String{N} or AsciiString{N}, as loaded, pass through, and any other string eltype is converted.

The ascii test becomes a loop over loaded ascii arrays, the Latin-1 block from 2, and mixed widths.

Patch 3 of 12
diff --git a/src/ASDF.jl b/src/ASDF.jl
index 98e75a6..20cea94 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -1832,7 +1832,11 @@ function NDArrayWrapper(array::AbstractArray; compression::Compression = C_Bzip2
 end
 Base.getindex(val::NDArrayWrapper) = val.array
 
-# Strings are stored as `ucs4`, as wide as the longest one, like a NumPy unicode array (at least 1).
+# Strings are stored as `ucs4`, as wide as the longest one, like a NumPy unicode array. Arrays of
+# `UCS4String{N}` or `AsciiString{N}`, as loaded, keep their datatype.
+ucs4_array(array::AbstractArray) = array
+ucs4_array(array::AbstractArray{UCS4String{N}}) where {N} = array
+ucs4_array(array::AbstractArray{AsciiString{N}}) where {N} = array
 function ucs4_array(array::AbstractArray{<:AbstractString})
     n = max(1, maximum(length, array; init = 0))
     out = Array{UCS4String{n}}(undef, size(array))
@@ -1846,11 +1850,9 @@ function ucs4_array(array::AbstractArray{<:AbstractString})
     return out
 end
 
-# Reject arrays that cannot be stored inline here, before `write_file` opens the file.
+# Store strings as `ucs4` and reject arrays that cannot be stored inline, before `write_file` opens the file.
 function _convert_tree_children(value::NDArrayWrapper, active)
-    if eltype(value.array) <: AbstractString && !(eltype(value.array) <: ASDFString)
-        value = NDArrayWrapper(ucs4_array(value.array), value.compression, value.inline, value.lz4_layout)
-    end
+    value = NDArrayWrapper(ucs4_array(value.array), value.compression, value.inline, value.lz4_layout)
     value.inline || return value
     array = value.array
     # The ndarray schema requires inline `data` to be a list, and Python `asdf` also refuses.
diff --git a/test/test-write.jl b/test/test-write.jl
index 5d6d7b1..5ce3b03 100644
--- a/test/test-write.jl
+++ b/test/test-write.jl
@@ -228,11 +228,17 @@ end
     file = ASDF.load_file(path)
     @test ASDF.compression_enums[file["labels"].lazy_block_headers.block_headers[1].compression] == ASDF.C_Zlib
     @test file["labels"][] == labels
-    # Loaded `ascii` arrays keep their datatype, inline too.
-    ASDF.write_file(path, Dict{Any, Any}("ascii" => ASDF.NDArrayWrapper([ASDF.AsciiString((0x61, 0x62)), ASDF.AsciiString((0x63, 0x00))]; inline = true)))
-    file = ASDF.load_file(path)
-    @test file["ascii"].datatype == ASDF.AsciiDatatype(2)
-    @test file["ascii"][] == ["ab", "c"]
+    # Loaded `ascii` arrays keep their datatype, inline too, and non-ASCII bytes round-trip in a block.
+    # Mixed-width string views have no single datatype, so they become `ucs4`.
+    ascii = [ASDF.AsciiString((0x61, 0x62)), ASDF.AsciiString((0x63, 0x00))]
+    latin1 = [ASDF.AsciiString((0x61, 0xe9))]
+    mixed = ASDF.UCS4String[ASDF.UCS4String(UInt32.(Tuple("a"))), ASDF.UCS4String(UInt32.(Tuple("bc")))]
+    for (array, inline, datatype) in ((ascii, true, ASDF.AsciiDatatype(2)), (latin1, false, ASDF.AsciiDatatype(2)), (mixed, false, ASDF.Ucs4Datatype(2)), (mixed, true, ASDF.Ucs4Datatype(2)))
+        ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(array; inline)))
+        file = ASDF.load_file(path)
+        @test file["a"].datatype == datatype
+        @test file["a"][] == array
+    end
     @test_throws "string \"a\\xffb\" contains the invalid character '\\xff', which has no UCS-4 code point" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\xffb"])))
 end
 
4. TaggedScalar arrays lose their tags

Problem. TaggedScalar <: AbstractString, so a wrapped array of tagged scalars was converted to plain ucs4. A YAML list of tagged scalars loads as a Vector{TaggedScalar}, so this is easy to reach:

ASDF.write_file(p, OrderedDict("a" => ASDF.NDArrayWrapper([ASDF.TaggedScalar("tag:example.org:x-1.0.0", "v1")])))
ASDF.load_file(p; extensions = true)["a"][]   # ["v1"], tag gone

Fix. Reject them.

Patch 4 of 12
diff --git a/src/ASDF.jl b/src/ASDF.jl
index 20cea94..2504e69 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -1841,6 +1841,7 @@ function ucs4_array(array::AbstractArray{<:AbstractString})
     n = max(1, maximum(length, array; init = 0))
     out = Array{UCS4String{n}}(undef, size(array))
     for (i, s) in enumerate(array)
+        s isa TaggedScalar && throw(ArgumentError("tagged scalar $(repr(s.value)) cannot be stored in an ndarray, which has no place for its tag"))
         codes = map(collect(s)) do c
             isvalid(c) || throw(ArgumentError("string $(repr(s)) contains the invalid character $(repr(c)), which has no UCS-4 code point"))
             UInt32(c)
diff --git a/test/test-write.jl b/test/test-write.jl
index 5ce3b03..018519b 100644
--- a/test/test-write.jl
+++ b/test/test-write.jl
@@ -240,6 +240,7 @@ end
         @test file["a"][] == array
     end
     @test_throws "string \"a\\xffb\" contains the invalid character '\\xff', which has no UCS-4 code point" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\xffb"])))
+    @test_throws "tagged scalar \"v1\" cannot be stored in an ndarray" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper([ASDF.TaggedScalar("tag:example.org:x-1.0.0", "v1")])))
 end
 
 @testset "ndarrays inside lists" begin
5. A trailing \0 is dropped on load

Problem. The reader treats trailing nulls as padding, as NumPy does, so they were lost silently:

wl(["a\0", "b"])   # ["a", "b"]

Fix. Reject such strings. The NDArrayWrapper docstring says so (12).

Patch 5 of 12
diff --git a/src/ASDF.jl b/src/ASDF.jl
index 2504e69..45a86fd 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -1847,6 +1847,8 @@ function ucs4_array(array::AbstractArray{<:AbstractString})
             UInt32(c)
         end
         out[i] = UCS4String{n}(ntuple(j -> get(codes, j, 0x00000000), n))
+        # Like NumPy, the reader drops trailing nulls as padding.
+        endswith(s, '\0') && throw(ArgumentError("string $(repr(s)) ends in a null character, which would be read back as padding"))
     end
     return out
 end
diff --git a/test/test-write.jl b/test/test-write.jl
index 018519b..4ba8280 100644
--- a/test/test-write.jl
+++ b/test/test-write.jl
@@ -241,6 +241,7 @@ end
     end
     @test_throws "string \"a\\xffb\" contains the invalid character '\\xff', which has no UCS-4 code point" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\xffb"])))
     @test_throws "tagged scalar \"v1\" cannot be stored in an ndarray" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper([ASDF.TaggedScalar("tag:example.org:x-1.0.0", "v1")])))
+    @test_throws "string \"a\\0\" ends in a null character, which would be read back as padding" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\0"])))
 end
 
 @testset "ndarrays inside lists" begin
6. Very long strings overflow the stack

Problem. Each element is an isbits value of 4n bytes, copied on the stack. The reader also overflows on element access at that width.

wl([repeat("x", 2_100_000)])   # StackOverflowError

Fix. Reject strings longer than max_string_width before the file is opened. I checked that a width of 2^16 still writes and reads back inside a task.

Patch 6 of 12
diff --git a/src/ASDF.jl b/src/ASDF.jl
index 45a86fd..21ad46c 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -1832,6 +1832,10 @@ function NDArrayWrapper(array::AbstractArray; compression::Compression = C_Bzip2
 end
 Base.getindex(val::NDArrayWrapper) = val.array
 
+# The widest string an ndarray element may hold. Elements are `isbits` values copied on the stack,
+# which a string of about 2 million characters overflows when written or read.
+const max_string_width = 2^16
+
 # Strings are stored as `ucs4`, as wide as the longest one, like a NumPy unicode array. Arrays of
 # `UCS4String{N}` or `AsciiString{N}`, as loaded, keep their datatype.
 ucs4_array(array::AbstractArray) = array
@@ -1839,6 +1843,7 @@ ucs4_array(array::AbstractArray{UCS4String{N}}) where {N} = array
 ucs4_array(array::AbstractArray{AsciiString{N}}) where {N} = array
 function ucs4_array(array::AbstractArray{<:AbstractString})
     n = max(1, maximum(length, array; init = 0))
+    n <= max_string_width || throw(ArgumentError("a string of $n characters is longer than the $max_string_width an ndarray element can hold"))
     out = Array{UCS4String{n}}(undef, size(array))
     for (i, s) in enumerate(array)
         s isa TaggedScalar && throw(ArgumentError("tagged scalar $(repr(s.value)) cannot be stored in an ndarray, which has no place for its tag"))
diff --git a/test/test-write.jl b/test/test-write.jl
index 4ba8280..98e1918 100644
--- a/test/test-write.jl
+++ b/test/test-write.jl
@@ -242,6 +242,7 @@ end
     @test_throws "string \"a\\xffb\" contains the invalid character '\\xff', which has no UCS-4 code point" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\xffb"])))
     @test_throws "tagged scalar \"v1\" cannot be stored in an ndarray" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper([ASDF.TaggedScalar("tag:example.org:x-1.0.0", "v1")])))
     @test_throws "string \"a\\0\" ends in a null character, which would be read back as padding" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\0"])))
+    @test_throws "is longer than the $(ASDF.max_string_width) an ndarray element can hold" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper([repeat("x", ASDF.max_string_width + 1)])))
 end
 
 @testset "ndarrays inside lists" begin
7. The writer and the inline reader check characters differently

Problem. The writer rejected "x\ud800" with an ArgumentError. An inline "x\ud800" loaded without error and failed only when iterated, and "x\xffy" threw a bare Base.InvalidCharError. The minimum-width rule was also written out twice.

Fix. One character check and one width rule, shared by the writer and the reader. Both inputs now raise the same ArgumentError at load time.

The writer's invalid-character message changes, since surrogates do have code points.

Patch 7 of 12
diff --git a/src/ASDF.jl b/src/ASDF.jl
index 21ad46c..acdd6b5 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -780,10 +780,13 @@ function inline_shape(data::AbstractVector)
     return shape
 end
 
+# The width of an `ascii` or `ucs4` datatype holding `strings`. At least 1, as in NumPy: all-empty
+# strings would otherwise give zero-byte elements and strides.
+string_width(strings) = max(1, maximum(length, strings; init = 0))
+
 # The datatype of inline data given without one, by the ndarray schema's inference rules.
 function inline_datatype(elements)
-    # At least 1, as in NumPy: all-empty strings would otherwise give zero-byte elements and strides.
-    any(x -> x isa AbstractString, elements) && return Ucs4Datatype(max(1, maximum(length ∘ string, elements)))
+    any(x -> x isa AbstractString, elements) && return Ucs4Datatype(string_width(string(x) for x in elements))
     any(x -> x isa Complex, elements) && return Datatype_complex128
     any(x -> x isa AbstractFloat, elements) && return Datatype_float64
     any(x -> x isa Integer && !(x isa Bool), elements) && return Datatype_int64
@@ -796,11 +799,17 @@ end
 inline_element(T::Type, x) = convert(T, x)
 function inline_element(T::Type{<:Tuple{Vararg{Union{UInt8, UInt32}}}}, s::AbstractString)
     n = fieldcount(T)
-    codes = [UInt32(c) for c in s]
+    codes = [string_code(eltype(T), c, s) for c in s]
     length(codes) <= n || throw(ArgumentError("inline string $(repr(s)) is longer than the $n characters of its datatype"))
-    eltype(T) === UInt8 && any(>(0x7f), codes) && throw(ArgumentError("inline string $(repr(s)) is not ASCII"))
     return T(ntuple(i -> i <= length(codes) ? codes[i] : 0, n))
 end
+
+# The code of character `c` of string `s` in an `ascii` (`UInt8`) or `ucs4` (`UInt32`) element.
+function string_code(T::Type{<:Union{UInt8, UInt32}}, c::AbstractChar, s::AbstractString)
+    isvalid(c) || throw(ArgumentError("string $(repr(s)) contains $(repr(c)), which is not a valid Unicode character"))
+    T === UInt8 && !isascii(c) && throw(ArgumentError("string $(repr(s)) is not ASCII"))
+    return T(c)
+end
 # Any string makes the whole array a string array, so numbers among the strings become strings too.
 inline_element(T::Type{<:Tuple{Vararg{Union{UInt8, UInt32}}}}, x::Real) = inline_element(T, string(x))
 
@@ -1842,15 +1851,12 @@ ucs4_array(array::AbstractArray) = array
 ucs4_array(array::AbstractArray{UCS4String{N}}) where {N} = array
 ucs4_array(array::AbstractArray{AsciiString{N}}) where {N} = array
 function ucs4_array(array::AbstractArray{<:AbstractString})
-    n = max(1, maximum(length, array; init = 0))
+    n = string_width(array)
     n <= max_string_width || throw(ArgumentError("a string of $n characters is longer than the $max_string_width an ndarray element can hold"))
     out = Array{UCS4String{n}}(undef, size(array))
     for (i, s) in enumerate(array)
         s isa TaggedScalar && throw(ArgumentError("tagged scalar $(repr(s.value)) cannot be stored in an ndarray, which has no place for its tag"))
-        codes = map(collect(s)) do c
-            isvalid(c) || throw(ArgumentError("string $(repr(s)) contains the invalid character $(repr(c)), which has no UCS-4 code point"))
-            UInt32(c)
-        end
+        codes = [string_code(UInt32, c, s) for c in s]
         out[i] = UCS4String{n}(ntuple(j -> get(codes, j, 0x00000000), n))
         # Like NumPy, the reader drops trailing nulls as padding.
         endswith(s, '\0') && throw(ArgumentError("string $(repr(s)) ends in a null character, which would be read back as padding"))
diff --git a/test/test-ndarray.jl b/test/test-ndarray.jl
index 444786e..e3452bd 100644
--- a/test/test-ndarray.jl
+++ b/test/test-ndarray.jl
@@ -376,6 +376,7 @@ end
     for (tree, message) in (
             ("a: !core/ndarray-1.1.0 {data: [abcd], datatype: [ascii, 3]}\n", "is longer than the 3 characters"),
             ("a: !core/ndarray-1.1.0 {data: [é], datatype: [ascii, 3]}\n", "is not ASCII"),
+            ("a: !core/ndarray-1.1.0 {data: [\"x\\ud800\"], datatype: [ucs4, 3]}\n", "is not a valid Unicode character"),
             # Masked values are not supported yet; see asdf-format/asdf-standard#451 and #452.
             ("a: !core/ndarray-1.1.0 {data: [1.0, null], datatype: float64}\n", "not supported yet"),
             ("a: !core/ndarray-1.1.0 {source: 0, datatype: float64, byteorder: little}\n", "needs `shape` and `datatype`"),
diff --git a/test/test-write.jl b/test/test-write.jl
index 98e1918..d319a40 100644
--- a/test/test-write.jl
+++ b/test/test-write.jl
@@ -239,7 +239,7 @@ end
         @test file["a"].datatype == datatype
         @test file["a"][] == array
     end
-    @test_throws "string \"a\\xffb\" contains the invalid character '\\xff', which has no UCS-4 code point" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\xffb"])))
+    @test_throws "string \"a\\xffb\" contains '\\xff', which is not a valid Unicode character" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\xffb"])))
     @test_throws "tagged scalar \"v1\" cannot be stored in an ndarray" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper([ASDF.TaggedScalar("tag:example.org:x-1.0.0", "v1")])))
     @test_throws "string \"a\\0\" ends in a null character, which would be read back as padding" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\0"])))
     @test_throws "is longer than the $(ASDF.max_string_width) an ndarray element can hold" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper([repeat("x", ASDF.max_string_width + 1)])))
8. The first inline load compiles a type per string length

Problem. inline_element padded with an Int 0, so each string length produced a new Tuple{UInt32..., Int64...} type to compile. Loading 50 inline strings of 1–8 characters plus one of 1000 took 20.5 s the first time.

Fix. Pad with the code type, so the tuple's type depends only on the width. The same load now takes 0.21 s.

Patch 8 of 12
diff --git a/src/ASDF.jl b/src/ASDF.jl
index acdd6b5..7c805a3 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -801,7 +801,8 @@ function inline_element(T::Type{<:Tuple{Vararg{Union{UInt8, UInt32}}}}, s::Abstr
     n = fieldcount(T)
     codes = [string_code(eltype(T), c, s) for c in s]
     length(codes) <= n || throw(ArgumentError("inline string $(repr(s)) is longer than the $n characters of its datatype"))
-    return T(ntuple(i -> i <= length(codes) ? codes[i] : 0, n))
+    # Pad with the code type, not `Int`, so the tuple's type depends only on the width.
+    return T(ntuple(i -> get(codes, i, zero(eltype(T))), n))
 end
 
 # The code of character `c` of string `s` in an `ascii` (`UInt8`) or `ucs4` (`UInt32`) element.
9. ucs4_array builds each string separately

Problem. For every string it made temporaries, called ntuple with a runtime length, and dispatched dynamically on UCS4String{n}. For 200k strings of 11–20 characters, ucs4_array took 200 ms and write_file with C_None took 219 ms.

Fix. Fill one UInt32 buffer and reinterpret it as strings. That takes 27 ms, and write_file 94 ms.

Patch 9 of 12
diff --git a/src/ASDF.jl b/src/ASDF.jl
index 7c805a3..a53210f 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -1854,15 +1854,16 @@ ucs4_array(array::AbstractArray{AsciiString{N}}) where {N} = array
 function ucs4_array(array::AbstractArray{<:AbstractString})
     n = string_width(array)
     n <= max_string_width || throw(ArgumentError("a string of $n characters is longer than the $max_string_width an ndarray element can hold"))
-    out = Array{UCS4String{n}}(undef, size(array))
-    for (i, s) in enumerate(array)
+    codes = zeros(UInt32, n, length(array))
+    for (j, s) in enumerate(array)
         s isa TaggedScalar && throw(ArgumentError("tagged scalar $(repr(s.value)) cannot be stored in an ndarray, which has no place for its tag"))
-        codes = [string_code(UInt32, c, s) for c in s]
-        out[i] = UCS4String{n}(ntuple(j -> get(codes, j, 0x00000000), n))
+        for (i, c) in enumerate(s)
+            codes[i, j] = string_code(UInt32, c, s)
+        end
         # Like NumPy, the reader drops trailing nulls as padding.
         endswith(s, '\0') && throw(ArgumentError("string $(repr(s)) ends in a null character, which would be read back as padding"))
     end
-    return out
+    return reshape(reinterpret(reshape, UCS4String{n}, codes), size(array))
 end
 
 # Store strings as `ucs4` and reject arrays that cannot be stored inline, before `write_file` opens the file.
10. Julia 1.10 writes string blocks byte by byte

Problem. On Julia 1.10, the compat floor, reinterpret(UInt8, ::Vector{UCS4String{N}}) reads each byte by copying out its whole element. So md5 and write cost O(bytes × width). With 10k labels plus one 1000-character string and C_None, the write took about 5 s on 1.10 (projected) and 0.46 s on 1.13.

Fix. Copy the code units into a matrix first. The same write takes 0.64 s on 1.10 and 0.20 s on 1.13.

Patch 10 of 12
diff --git a/src/ASDF.jl b/src/ASDF.jl
index a53210f..f5d2a3e 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -1990,6 +1990,16 @@ end
 # record for alignment and the block packs them (see `itemsize`), so padded records are packed field
 # by field.
 block_bytes(array::AbstractVector) = reinterpret(UInt8, array)
+# Julia 1.10 reads each byte of a `reinterpret`ed array by copying out its whole element, which is
+# slow for wide strings, so their code units are copied out first.
+function block_bytes(array::AbstractVector{T}) where {T <: ASDFString}
+    codes = fieldtype(T, :codes)
+    units = Matrix{eltype(codes)}(undef, fieldcount(codes), length(array))
+    for (j, s) in enumerate(array)
+        units[:, j] .= s.codes
+    end
+    return reinterpret(UInt8, vec(units))
+end
 function block_bytes(array::AbstractVector{T}) where {T <: NamedTuple}
     dt = infer_asdf_datatype(T)
     itemsize(dt) == sizeof(T) && return reinterpret(UInt8, array)
11. The tests don't check inline storage

Problem. If the re-wrap in _convert_tree_children dropped value.inline, every "inline" array would be written as a block and the testset would still pass. The rejection tests also didn't check that the existing file is left intact.

Fix. Keep each array's width next to it in arrays, and assert where each array is stored. The eltype check is dropped because the datatype check implies it.

The rejection tests from 4–7 become one loop that also checks the file is left unchanged.

Patch 11 of 12
diff --git a/test/test-write.jl b/test/test-write.jl
index d319a40..3e82061 100644
--- a/test/test-write.jl
+++ b/test/test-write.jl
@@ -208,16 +208,16 @@ end
     # `escapes` holds strings that YAML.jl's string printer can't round-trip: NUL ends its YAML stream,
     # Julia's short `\u` escapes are invalid YAML, and block literals lose leading spaces and CRs.
     escapes = ["\"\0x", "co\u00adop", "don\u0092t", "\u85", "\U1f3f4\Ue0067\Ue0062\Ue0073\Ue0063\Ue0074\Ue007f", "  a\nb", "a\r\nb", " \"x", "x ", "\\n", "\t\u2028\ufeff"]
-    arrays = Dict{Any, Any}("matrix" => ["" "right"; "left" ""], "unicode" => ["é", "🎉x", "true", "1", "a\nb", "\"q\""], "escapes" => escapes, "empty_strings" => ["", ""], "no_strings" => String[], "zero_dimensional" => fill("abc"), "substrings" => split("a bc"))
-    widths = Dict("matrix" => 5, "unicode" => 4, "escapes" => 7, "empty_strings" => 1, "no_strings" => 1, "zero_dimensional" => 3, "substrings" => 2)
+    # Each array with its `ucs4` width.
+    arrays = Dict{Any, Any}("matrix" => (["" "right"; "left" ""], 5), "unicode" => (["é", "🎉x", "true", "1", "a\nb", "\"q\""], 4), "escapes" => (escapes, 7), "empty_strings" => (["", ""], 1), "no_strings" => (String[], 1), "zero_dimensional" => (fill("abc"), 3), "substrings" => (split("a bc"), 2))
     for inline in (false, true)
         written = inline ? delete!(copy(arrays), "zero_dimensional") : arrays
-        ASDF.write_file(path, Dict{Any, Any}("list" => ["a", "b"], (key => ASDF.NDArrayWrapper(array; inline) for (key, array) in written)...))
+        ASDF.write_file(path, Dict{Any, Any}("list" => ["a", "b"], (key => ASDF.NDArrayWrapper(array; inline) for (key, (array, _)) in written)...))
         file = ASDF.load_file(path)
         @test file["list"] == ["a", "b"]
-        for (key, array) in written
-            @test file[key].datatype == ASDF.Ucs4Datatype(widths[key])
-            @test eltype(file[key][]) == ASDF.UCS4String{widths[key]}
+        for (key, (array, width)) in written
+            @test (file[key].source === nothing) == inline
+            @test file[key].datatype == ASDF.Ucs4Datatype(width)
             @test size(file[key][]) == size(array)
             @test file[key][] == array
         end
@@ -239,10 +239,17 @@ end
         @test file["a"].datatype == datatype
         @test file["a"][] == array
     end
-    @test_throws "string \"a\\xffb\" contains '\\xff', which is not a valid Unicode character" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\xffb"])))
-    @test_throws "tagged scalar \"v1\" cannot be stored in an ndarray" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper([ASDF.TaggedScalar("tag:example.org:x-1.0.0", "v1")])))
-    @test_throws "string \"a\\0\" ends in a null character, which would be read back as padding" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\0"])))
-    @test_throws "is longer than the $(ASDF.max_string_width) an ndarray element can hold" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper([repeat("x", ASDF.max_string_width + 1)])))
+    # Strings the reader can't return as written are rejected before `write_file` opens the file.
+    contents = read(path)
+    for (array, message) in (
+            (["a\xffb"], "string \"a\\xffb\" contains '\\xff', which is not a valid Unicode character"),
+            (["a\0"], "string \"a\\0\" ends in a null character, which would be read back as padding"),
+            ([ASDF.TaggedScalar("tag:example.org:x-1.0.0", "v1")], "tagged scalar \"v1\" cannot be stored in an ndarray"),
+            ([repeat("x", ASDF.max_string_width + 1)], "is longer than the $(ASDF.max_string_width) an ndarray element can hold"),
+        )
+        @test_throws message ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(array)))
+        @test read(path) == contents
+    end
 end
 
 @testset "ndarrays inside lists" begin
12. Docs and comments

Problem. docs/src/index.md listed only arrays of numbers. The NDArrayWrapper docstring didn't cover loaded string views or trailing nulls. A test comment still said YAML.jl reads inline elements back only as booleans, Int64s and floats.

Fix. Mention string arrays in index.md, cover loaded string views and trailing nulls in the docstring, and correct the test comment.

Patch 12 of 12
diff --git a/docs/src/index.md b/docs/src/index.md
index 2dd91c5..9cba192 100644
--- a/docs/src/index.md
+++ b/docs/src/index.md
@@ -4,7 +4,7 @@ A new [Advanced Scientific Data Format (ASDF)](https://asdf-standard.readthedocs
 
 ## Introduction
 
-The ASDF file format is based on the human-readable [YAML](http://yaml.org/) standard, extended with efficient binary blocks to store array data. Basic arithmetic types (`Bool`, `Int`, `Float`) and `String` types are supported out of the box, as are arrays of real and complex numbers. Other types (structures) need to be declared to be supported.
+The ASDF file format is based on the human-readable [YAML](http://yaml.org/) standard, extended with efficient binary blocks to store array data. Basic arithmetic types (`Bool`, `Int`, `Float`) and `String` types are supported out of the box, as are arrays of real and complex numbers and of strings. Other types (structures) need to be declared to be supported.
 
 ASDF supports arbitrary array strides, both C (Python) and Fortran (Julia) memory layouts, as well as compression. The YAML metadata can contain arbitrary information corresponding to scalars, arrays, or dictionaries.
 
diff --git a/src/ASDF.jl b/src/ASDF.jl
index f5d2a3e..204cf2f 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -1820,7 +1820,7 @@ _convert_tree_children(value::ASDFLibrary, active) = value
 
 A write-side wrapper around a Julia array that carries compression and layout options. Used as the value type when building a document dict for [`ASDF.write_file`](@ref).
 
-An array of strings is written as a `ucs4` array as wide as its longest string (at least 1 character), like a NumPy unicode array.
+An array of strings is written as a `ucs4` array as wide as its longest string (at least 1 character), like a NumPy unicode array. Since trailing nulls read back as padding, no string may end in one. Arrays of [`UCS4String{N}`](@ref UCS4String) or [`AsciiString{N}`](@ref AsciiString), as loaded from a file, keep their datatype.
 
 Parameter     | Default   | Description                                                                                                        |
 | :---------- | :-------- | :----------------------------------------------------------------------------------------------------------------- |
diff --git a/test/test-write.jl b/test/test-write.jl
index 3e82061..bab9e89 100644
--- a/test/test-write.jl
+++ b/test/test-write.jl
@@ -190,8 +190,9 @@ end
 
 @testset "reject arrays that cannot be stored inline" begin
     # The ndarray schema requires inline data to be a list, so a 0-dimensional array has no inline form.
-    # YAML.jl reads inline elements back only as booleans, `Int64`s, and floats. The arrays are rejected
-    # before `write_file` opens the file, which leaves an existing file intact.
+    # YAML.jl reads inline elements back only as booleans, `Int64`s, floats, and strings, and the reader
+    # accepts only ASCII `ascii` strings. The arrays are rejected before `write_file` opens the file, which
+    # leaves an existing file intact.
     path = joinpath(mktempdir(; cleanup = true), "existing.asdf")
     ASDF.write_file(path, Dict{Any, Any}("kept" => 1))
     contents = read(path)

For you to decide

  • max_string_width = 2^16 is my pick. Around 2M characters overflows the stack, so any limit well below that works.
  • Strings ending in \0 are rejected. NumPy strips trailing nulls silently. Matching NumPy would turn the check in 5 into a docstring note.
  • Tabs and newlines are written as \u0009 and \u000a. Short escapes (\t, \n) would read better but need a small lookup table.

Not addressed (already on main)

  • Plain metadata strings and mapping keys still go through YAML.jl's string printer, with the same failures as in 1.
  • YAML.write(::ASDFFile) skips _convert_tree, so floats lose the e+ exponent sign and string wrappers throw KeyError.
  • An AsciiString holding bytes above 0x7f is isequal to a String but hashes differently (seen on 1.13).
  • C_Blosc2 fails after write_file has already replaced the target file.
  • UCS4String and AsciiString lack isvalid(s) and reverse.
  • On Julia 1.10, other wide element types (NTuple, records) are still read byte by byte when written.
Combined diff of all 12 patches
diff --git a/docs/src/index.md b/docs/src/index.md
index 2dd91c5..9cba192 100644
--- a/docs/src/index.md
+++ b/docs/src/index.md
@@ -4,7 +4,7 @@ A new [Advanced Scientific Data Format (ASDF)](https://asdf-standard.readthedocs
 
 ## Introduction
 
-The ASDF file format is based on the human-readable [YAML](http://yaml.org/) standard, extended with efficient binary blocks to store array data. Basic arithmetic types (`Bool`, `Int`, `Float`) and `String` types are supported out of the box, as are arrays of real and complex numbers. Other types (structures) need to be declared to be supported.
+The ASDF file format is based on the human-readable [YAML](http://yaml.org/) standard, extended with efficient binary blocks to store array data. Basic arithmetic types (`Bool`, `Int`, `Float`) and `String` types are supported out of the box, as are arrays of real and complex numbers and of strings. Other types (structures) need to be declared to be supported.
 
 ASDF supports arbitrary array strides, both C (Python) and Fortran (Julia) memory layouts, as well as compression. The YAML metadata can contain arbitrary information corresponding to scalars, arrays, or dictionaries.
 
diff --git a/src/ASDF.jl b/src/ASDF.jl
index d5631d8..204cf2f 100644
--- a/src/ASDF.jl
+++ b/src/ASDF.jl
@@ -780,10 +780,13 @@ function inline_shape(data::AbstractVector)
     return shape
 end
 
+# The width of an `ascii` or `ucs4` datatype holding `strings`. At least 1, as in NumPy: all-empty
+# strings would otherwise give zero-byte elements and strides.
+string_width(strings) = max(1, maximum(length, strings; init = 0))
+
 # The datatype of inline data given without one, by the ndarray schema's inference rules.
 function inline_datatype(elements)
-    # At least 1, as in NumPy: all-empty strings would otherwise give zero-byte elements and strides.
-    any(x -> x isa AbstractString, elements) && return Ucs4Datatype(max(1, maximum(length ∘ string, elements)))
+    any(x -> x isa AbstractString, elements) && return Ucs4Datatype(string_width(string(x) for x in elements))
     any(x -> x isa Complex, elements) && return Datatype_complex128
     any(x -> x isa AbstractFloat, elements) && return Datatype_float64
     any(x -> x isa Integer && !(x isa Bool), elements) && return Datatype_int64
@@ -796,10 +799,17 @@ end
 inline_element(T::Type, x) = convert(T, x)
 function inline_element(T::Type{<:Tuple{Vararg{Union{UInt8, UInt32}}}}, s::AbstractString)
     n = fieldcount(T)
-    codes = [UInt32(c) for c in s]
+    codes = [string_code(eltype(T), c, s) for c in s]
     length(codes) <= n || throw(ArgumentError("inline string $(repr(s)) is longer than the $n characters of its datatype"))
-    eltype(T) === UInt8 && any(>(0x7f), codes) && throw(ArgumentError("inline string $(repr(s)) is not ASCII"))
-    return T(ntuple(i -> i <= length(codes) ? codes[i] : 0, n))
+    # Pad with the code type, not `Int`, so the tuple's type depends only on the width.
+    return T(ntuple(i -> get(codes, i, zero(eltype(T))), n))
+end
+
+# The code of character `c` of string `s` in an `ascii` (`UInt8`) or `ucs4` (`UInt32`) element.
+function string_code(T::Type{<:Union{UInt8, UInt32}}, c::AbstractChar, s::AbstractString)
+    isvalid(c) || throw(ArgumentError("string $(repr(s)) contains $(repr(c)), which is not a valid Unicode character"))
+    T === UInt8 && !isascii(c) && throw(ArgumentError("string $(repr(s)) is not ASCII"))
+    return T(c)
 end
 # Any string makes the whole array a string array, so numbers among the strings become strings too.
 inline_element(T::Type{<:Tuple{Vararg{Union{UInt8, UInt32}}}}, x::Real) = inline_element(T, string(x))
@@ -1366,10 +1376,33 @@ struct YAMLScalar
 end
 YAML._print(io::IO, val::YAMLScalar, level::Int = 0, ignore_level::Bool = false) = println(io, val.text)
 
-# Rewrite inline ndarray data so every float is emitted in YAML-1.1-compliant form. Document
-# metadata gets the same treatment from `_convert_tree`. The data nests one list per dimension, the
-# first `shape` (last Julia) dimension outermost.
+# A string as a YAML double-quoted scalar with every unprintable character, line breaks included,
+# escaped at full width. YAML.jl's own string printer cannot round-trip arbitrary strings: Julia's
+# short escapes (`\u85`) are invalid YAML, and its block literals drop leading spaces and CRs and
+# write NUL raw, which ends the YAML stream.
+function yaml_quoted_string(s::String)
+    return sprint() do io
+        print(io, '"')
+        for c in s
+            if c == '"' || c == '\\'
+                print(io, '\\', c)
+            elseif isprint(c)
+                print(io, c)
+            elseif c <= '\uffff'
+                print(io, "\\u", string(UInt32(c); base = 16, pad = 4))
+            else
+                print(io, "\\U", string(UInt32(c); base = 16, pad = 8))
+            end
+        end
+        print(io, '"')
+    end
+end
+
+# Rewrite inline ndarray data so every float is emitted in YAML-1.1-compliant form, and every
+# string round-trips. Document metadata floats get the same treatment from `_convert_tree`. The data
+# nests one list per dimension, the first `shape` (last Julia) dimension outermost.
 yaml_compliant(val::AbstractFloat) = YAMLScalar(yaml_float_string(val))
+yaml_compliant(val::ASDFString) = YAMLScalar(yaml_quoted_string(String(val)))
 yaml_compliant(val::AbstractVector) = map(yaml_compliant, val)
 yaml_compliant(val::AbstractArray) = map(yaml_compliant, eachslice(val; dims = ndims(val)))
 yaml_compliant(val) = val
@@ -1787,7 +1820,7 @@ _convert_tree_children(value::ASDFLibrary, active) = value
 
 A write-side wrapper around a Julia array that carries compression and layout options. Used as the value type when building a document dict for [`ASDF.write_file`](@ref).
 
-An array of strings is written as a `ucs4` array as wide as its longest string (at least 1 character), like a NumPy unicode array.
+An array of strings is written as a `ucs4` array as wide as its longest string (at least 1 character), like a NumPy unicode array. Since trailing nulls read back as padding, no string may end in one. Arrays of [`UCS4String{N}`](@ref UCS4String) or [`AsciiString{N}`](@ref AsciiString), as loaded from a file, keep their datatype.
 
 Parameter     | Default   | Description                                                                                                        |
 | :---------- | :-------- | :----------------------------------------------------------------------------------------------------------------- |
@@ -1809,25 +1842,33 @@ function NDArrayWrapper(array::AbstractArray; compression::Compression = C_Bzip2
 end
 Base.getindex(val::NDArrayWrapper) = val.array
 
-# Strings are stored as `ucs4`, as wide as the longest one, like a NumPy unicode array (at least 1).
+# The widest string an ndarray element may hold. Elements are `isbits` values copied on the stack,
+# which a string of about 2 million characters overflows when written or read.
+const max_string_width = 2^16
+
+# Strings are stored as `ucs4`, as wide as the longest one, like a NumPy unicode array. Arrays of
+# `UCS4String{N}` or `AsciiString{N}`, as loaded, keep their datatype.
+ucs4_array(array::AbstractArray) = array
+ucs4_array(array::AbstractArray{UCS4String{N}}) where {N} = array
+ucs4_array(array::AbstractArray{AsciiString{N}}) where {N} = array
 function ucs4_array(array::AbstractArray{<:AbstractString})
-    n = max(1, maximum(length, array; init = 0))
-    out = Array{UCS4String{n}}(undef, size(array))
-    for (i, s) in enumerate(array)
-        codes = map(collect(s)) do c
-            isvalid(c) || throw(ArgumentError("string $(repr(s)) contains the invalid character $(repr(c)), which has no UCS-4 code point"))
-            UInt32(c)
+    n = string_width(array)
+    n <= max_string_width || throw(ArgumentError("a string of $n characters is longer than the $max_string_width an ndarray element can hold"))
+    codes = zeros(UInt32, n, length(array))
+    for (j, s) in enumerate(array)
+        s isa TaggedScalar && throw(ArgumentError("tagged scalar $(repr(s.value)) cannot be stored in an ndarray, which has no place for its tag"))
+        for (i, c) in enumerate(s)
+            codes[i, j] = string_code(UInt32, c, s)
         end
-        out[i] = UCS4String{n}(ntuple(j -> get(codes, j, 0x00000000), n))
+        # Like NumPy, the reader drops trailing nulls as padding.
+        endswith(s, '\0') && throw(ArgumentError("string $(repr(s)) ends in a null character, which would be read back as padding"))
     end
-    return out
+    return reshape(reinterpret(reshape, UCS4String{n}, codes), size(array))
 end
 
-# Reject arrays that cannot be stored inline here, before `write_file` opens the file.
+# Store strings as `ucs4` and reject arrays that cannot be stored inline, before `write_file` opens the file.
 function _convert_tree_children(value::NDArrayWrapper, active)
-    if eltype(value.array) <: AbstractString && !(eltype(value.array) <: ASDFString)
-        value = NDArrayWrapper(ucs4_array(value.array), value.compression, value.inline, value.lz4_layout)
-    end
+    value = NDArrayWrapper(ucs4_array(value.array), value.compression, value.inline, value.lz4_layout)
     value.inline || return value
     array = value.array
     # The ndarray schema requires inline `data` to be a list, and Python `asdf` also refuses.
@@ -1835,6 +1876,8 @@ function _convert_tree_children(value::NDArrayWrapper, active)
     # Inline elements are YAML scalars, which YAML.jl reads back only as booleans, `Int64`s, floats, and strings.
     eltype(array) <: Union{Integer, AbstractFloat, ASDFString} || throw(ArgumentError("an array of $(eltype(array)) cannot be stored inline; use `inline = false`"))
     eltype(array) <: UInt64 && any(>(typemax(Int64)), array) && throw(ArgumentError("`UInt64` values above `typemax(Int64)` cannot be stored inline; use `inline = false`"))
+    # The inline reader accepts only ASCII `ascii` strings.
+    eltype(array) <: AsciiString && !all(isascii, array) && throw(ArgumentError("`ascii` strings with non-ASCII bytes cannot be stored inline; use `inline = false`"))
     return value
 end
 
@@ -1910,7 +1953,7 @@ function YAML._print(io::IO, val::NDArrayWrapper, level::Int = 0, ignore_level::
     shape = collect(Int64, reverse(size(val.array)))
     if val.inline
         ndarray = OrderedDict(
-            # Route inline data through the float pass so its elements get YAML-1.1 exponents too.
+            # Route inline data through `yaml_compliant` so floats get YAML-1.1 exponents and strings round-trip.
             # `collect` first: `eachslice` misindexes views of views with `Base.IdentityUnitRange` indices.
             :data => yaml_compliant(collect(val.array)),
             :shape => shape,
@@ -1947,6 +1990,16 @@ end
 # record for alignment and the block packs them (see `itemsize`), so padded records are packed field
 # by field.
 block_bytes(array::AbstractVector) = reinterpret(UInt8, array)
+# Julia 1.10 reads each byte of a `reinterpret`ed array by copying out its whole element, which is
+# slow for wide strings, so their code units are copied out first.
+function block_bytes(array::AbstractVector{T}) where {T <: ASDFString}
+    codes = fieldtype(T, :codes)
+    units = Matrix{eltype(codes)}(undef, fieldcount(codes), length(array))
+    for (j, s) in enumerate(array)
+        units[:, j] .= s.codes
+    end
+    return reinterpret(UInt8, vec(units))
+end
 function block_bytes(array::AbstractVector{T}) where {T <: NamedTuple}
     dt = infer_asdf_datatype(T)
     itemsize(dt) == sizeof(T) && return reinterpret(UInt8, array)
diff --git a/test/test-ndarray.jl b/test/test-ndarray.jl
index 444786e..e3452bd 100644
--- a/test/test-ndarray.jl
+++ b/test/test-ndarray.jl
@@ -376,6 +376,7 @@ end
     for (tree, message) in (
             ("a: !core/ndarray-1.1.0 {data: [abcd], datatype: [ascii, 3]}\n", "is longer than the 3 characters"),
             ("a: !core/ndarray-1.1.0 {data: [é], datatype: [ascii, 3]}\n", "is not ASCII"),
+            ("a: !core/ndarray-1.1.0 {data: [\"x\\ud800\"], datatype: [ucs4, 3]}\n", "is not a valid Unicode character"),
             # Masked values are not supported yet; see asdf-format/asdf-standard#451 and #452.
             ("a: !core/ndarray-1.1.0 {data: [1.0, null], datatype: float64}\n", "not supported yet"),
             ("a: !core/ndarray-1.1.0 {source: 0, datatype: float64, byteorder: little}\n", "needs `shape` and `datatype`"),
diff --git a/test/test-write.jl b/test/test-write.jl
index facdc0d..bab9e89 100644
--- a/test/test-write.jl
+++ b/test/test-write.jl
@@ -190,12 +190,13 @@ end
 
 @testset "reject arrays that cannot be stored inline" begin
     # The ndarray schema requires inline data to be a list, so a 0-dimensional array has no inline form.
-    # YAML.jl reads inline elements back only as booleans, `Int64`s, and floats. The arrays are rejected
-    # before `write_file` opens the file, which leaves an existing file intact.
+    # YAML.jl reads inline elements back only as booleans, `Int64`s, floats, and strings, and the reader
+    # accepts only ASCII `ascii` strings. The arrays are rejected before `write_file` opens the file, which
+    # leaves an existing file intact.
     path = joinpath(mktempdir(; cleanup = true), "existing.asdf")
     ASDF.write_file(path, Dict{Any, Any}("kept" => 1))
     contents = read(path)
-    for array in (fill(1.5), ComplexF64[1 + 2im], [(0x61, 0x62)], [(a = 1, b = 2.0)], UInt64[typemax(UInt64)])
+    for array in (fill(1.5), ComplexF64[1 + 2im], [(0x61, 0x62)], [(a = 1, b = 2.0)], UInt64[typemax(UInt64)], [ASDF.AsciiString((0x61, 0xe9))])
         @test_throws "cannot be stored inline; use `inline = false`" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(array; inline = true)))
         @test read(path) == contents
     end
@@ -205,16 +206,19 @@ end
     # Like a NumPy unicode array, a wrapped string array is a `ucs4` ndarray as wide as its longest string,
     # and at least 1 wide. A plain vector of strings stays a YAML list.
     path = joinpath(mktempdir(; cleanup = true), "strings.asdf")
-    arrays = Dict{Any, Any}("matrix" => ["" "right"; "left" ""], "unicode" => ["é", "🎉x", "true", "1", "a\nb", "\"q\""], "empty_strings" => ["", ""], "no_strings" => String[], "zero_dimensional" => fill("abc"), "substrings" => split("a bc"))
-    widths = Dict("matrix" => 5, "unicode" => 4, "empty_strings" => 1, "no_strings" => 1, "zero_dimensional" => 3, "substrings" => 2)
+    # `escapes` holds strings that YAML.jl's string printer can't round-trip: NUL ends its YAML stream,
+    # Julia's short `\u` escapes are invalid YAML, and block literals lose leading spaces and CRs.
+    escapes = ["\"\0x", "co\u00adop", "don\u0092t", "\u85", "\U1f3f4\Ue0067\Ue0062\Ue0073\Ue0063\Ue0074\Ue007f", "  a\nb", "a\r\nb", " \"x", "x ", "\\n", "\t\u2028\ufeff"]
+    # Each array with its `ucs4` width.
+    arrays = Dict{Any, Any}("matrix" => (["" "right"; "left" ""], 5), "unicode" => (["é", "🎉x", "true", "1", "a\nb", "\"q\""], 4), "escapes" => (escapes, 7), "empty_strings" => (["", ""], 1), "no_strings" => (String[], 1), "zero_dimensional" => (fill("abc"), 3), "substrings" => (split("a bc"), 2))
     for inline in (false, true)
         written = inline ? delete!(copy(arrays), "zero_dimensional") : arrays
-        ASDF.write_file(path, Dict{Any, Any}("list" => ["a", "b"], (key => ASDF.NDArrayWrapper(array; inline) for (key, array) in written)...))
+        ASDF.write_file(path, Dict{Any, Any}("list" => ["a", "b"], (key => ASDF.NDArrayWrapper(array; inline) for (key, (array, _)) in written)...))
         file = ASDF.load_file(path)
         @test file["list"] == ["a", "b"]
-        for (key, array) in written
-            @test file[key].datatype == ASDF.Ucs4Datatype(widths[key])
-            @test eltype(file[key][]) == ASDF.UCS4String{widths[key]}
+        for (key, (array, width)) in written
+            @test (file[key].source === nothing) == inline
+            @test file[key].datatype == ASDF.Ucs4Datatype(width)
             @test size(file[key][]) == size(array)
             @test file[key][] == array
         end
@@ -225,12 +229,28 @@ end
     file = ASDF.load_file(path)
     @test ASDF.compression_enums[file["labels"].lazy_block_headers.block_headers[1].compression] == ASDF.C_Zlib
     @test file["labels"][] == labels
-    # Loaded `ascii` arrays keep their datatype, inline too.
-    ASDF.write_file(path, Dict{Any, Any}("ascii" => ASDF.NDArrayWrapper([ASDF.AsciiString((0x61, 0x62)), ASDF.AsciiString((0x63, 0x00))]; inline = true)))
-    file = ASDF.load_file(path)
-    @test file["ascii"].datatype == ASDF.AsciiDatatype(2)
-    @test file["ascii"][] == ["ab", "c"]
-    @test_throws "string \"a\\xffb\" contains the invalid character '\\xff', which has no UCS-4 code point" ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(["a\xffb"])))
+    # Loaded `ascii` arrays keep their datatype, inline too, and non-ASCII bytes round-trip in a block.
+    # Mixed-width string views have no single datatype, so they become `ucs4`.
+    ascii = [ASDF.AsciiString((0x61, 0x62)), ASDF.AsciiString((0x63, 0x00))]
+    latin1 = [ASDF.AsciiString((0x61, 0xe9))]
+    mixed = ASDF.UCS4String[ASDF.UCS4String(UInt32.(Tuple("a"))), ASDF.UCS4String(UInt32.(Tuple("bc")))]
+    for (array, inline, datatype) in ((ascii, true, ASDF.AsciiDatatype(2)), (latin1, false, ASDF.AsciiDatatype(2)), (mixed, false, ASDF.Ucs4Datatype(2)), (mixed, true, ASDF.Ucs4Datatype(2)))
+        ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(array; inline)))
+        file = ASDF.load_file(path)
+        @test file["a"].datatype == datatype
+        @test file["a"][] == array
+    end
+    # Strings the reader can't return as written are rejected before `write_file` opens the file.
+    contents = read(path)
+    for (array, message) in (
+            (["a\xffb"], "string \"a\\xffb\" contains '\\xff', which is not a valid Unicode character"),
+            (["a\0"], "string \"a\\0\" ends in a null character, which would be read back as padding"),
+            ([ASDF.TaggedScalar("tag:example.org:x-1.0.0", "v1")], "tagged scalar \"v1\" cannot be stored in an ndarray"),
+            ([repeat("x", ASDF.max_string_width + 1)], "is longer than the $(ASDF.max_string_width) an ndarray element can hold"),
+        )
+        @test_throws message ASDF.write_file(path, Dict{Any, Any}("a" => ASDF.NDArrayWrapper(array)))
+        @test read(path) == contents
+    end
 end
 
 @testset "ndarrays inside lists" begin

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants