Matrix support ?

[eddelbuettel]

I am contemplating doing something where I may need / want (dense) matrix support, and I have been thinking that an approach similar to what R does (i.e. have it be a vector that happens to have a dimension attribute) may be an option by sticking a dimension vector into the void* private_data slot. This seems both 'hackish' yet somewhat obvious.

Or am I missing existing prior work elsewhere? Has there really not been any other work where matrices are being passed around via the arrow interface? Thanks in advance for any pointers, or even just a blank 'you are nuts and here is why ...'.

[lidavidm]

(1) For this sort of thing, you should use an extension type as that is the standard way to "layer" new semantics and type metadata onto an existing Arrow type. (2), does the "fixed shape tensor" extension type suit your needs?

[eddelbuettel]

Tensors had come up in a past life, albeit with the caveat that what was out there was 'early and raw'.

But thanks for sending me on my way here. I will take a good look at extension types. No issue here, so closing. Thanks again!

[eddelbuettel]

Thanks for the re-open, Dewey. After spitballing the two references by David it seems we could do something here in nanoarrow around its interfaces. Thoughts?

[paleolimbot]

Reopening because matrix support is a great idea! Unfortunately the column-majorness of matrices in R makes some of the ways we can deal with this not be zero copy, but I think we should still track this even if we don't get to it right away.

For a matrix column in an Array, I think the natural mapping would be a fixed_size_list() (i.e., each row is a fixed size list of whatever matrix type we're dealing with). Maybe:

mat <- matrix(1:6, ncol = 2, byrow = TRUE)
infer_nanoarrow_type(mat)  # would be na_fixed_size_list(na_int32())
array <- as_nanoarrow_array(mat)  # would be logically [[1, 2], [3, 4], [5, 6]]

# We already have a mapping for list-like things, so the default would still come back as a list()
convert_array(array). # -> list(1:2, 3:4, 5:6)

# ...but we can request a type from convert_array(), so we can implement the conversion
convert_array(array, matrix(integer())  # -> matrix(1:6, ncol = 2, byrow = TRUE)

The "fixed shape tensor" extension would perhaps map to an array(), but unfortunately an array() with two dimensions is indistinguishable from a matrix() in all recent Rs. This is also less common to have an array column, so perhaps we could punt on that for now.

[eddelbuettel]

At first glance, that seems expensive. My (initial) thinking was that i) vectors are great, and (nano)array arrays are vectors (in the simple instance of a contiguous array) and ii) most linear algebra consumers take such a vector and merrily form a matrix from it, often zero-copy. So from mat <- matrix(1:6, ncol =2, byrow = TRUE) I really want the vec <- as.vector(mat) and wondering if there is a good way to 'also' attach the the dim <- c(3,2) part in way that makes sense. Metadata may be a way that is simpler and more robust that a schema-in-private-data, my initial idea.

But you all know the wider arrow landscape better than I do so please shoot holes into this at earliest convenience...

[paleolimbot]

Ah! For that, we definitely do want the extension type. What we have there would be:

mat_list <- list(matrix(1:6, ncol = 2, byrow = TRUE))
as_nanoarrow_array(mat_list, na_fixed_size_tensor(na_int32(), shape = c(2L, 3L), permutation = c(1L, 0L))

...which would be zero copy (limited to a "single element", which is, of course, a lot of elements they're just in the matrix part). That's impossible enough to type that maybe we just want a function for that.

[eddelbuettel]

Thanks for #692, @paleolimbot. It is an excellent first step and 'proof of the pudding'. I am now trying to complement it from the other side closer to my initial use case / idea and idea by starting from in C(++), i.e. trying to create a proper struct containing such a fixed_sized_list wrapping a matrix. Needless to say a naive

ArrowSchemaInitFromType(sch->children[1], NANOARROW_TYPE_FIXED_SIZE_LIST)

hits the wall quite quickly. Do you have plans to complement #692 either here in src/ and/or in, say, the ADBC packages?

[paleolimbot]

ArrowSchemaInitFromType(sch->children[1], NANOARROW_TYPE_FIXED_SIZE_LIST)

I think today you need ArowSchemaInit() + ArrowSchemaSetTypeFixedSize(sch->children[1], NANOARROW_TYPE_FIXED_SIZE_LIST, 123xx);

Do you have plans to complement #692 either here in src/ and/or in, say, the ADBC packages?

The ArrowArrayAppenders are pretty good here already; I would like to make ArrowArrayViewGetListElementUnsafe() to help with the consuming side (which helps the recursive composing when reading arbitrarily nested things).

[eddelbuettel]

That looks promising. Was wondering where the size element would enter -- this makes sense. (Also, the spec of +w:3 does not give the representation aka storage type.)

As for ArrowArrayAppend I cobbled together a 'local' one which copies vectors 'whole' in one memcpy() step (taking arguments for length and element size and along with an optional validity buffer) but I hear you -- the nesting here will require an extra step.

[paleolimbot]

Oh, sorry (you would follow that with ArrowSchemaSetType(sch->children[1]->children[0], NANOARROW_XXX); to set the list item type.

As for ArrowArrayAppend

I think what you've done is the most efficient if your backend is already vectorized.

If your child elements are sitting in something like JSON, you can use ArrowArrayAppendXXXX(fixed_size_list_array->children[0], for_each_child_element); ArrowArrayFinishElement(fixed_size_list_array); for as many items in the outer list.

[eddelbuettel]

Thanks -- got back to this now and got it working, with a little trial and error. Even emits components with the JSON printer I borrowed from your test code. A test function called with an int vector and a double vector intented for a two column matrix comes out fine as in your R example.