How to deserialize RecordBatch

[slinkydeveloper]

Hi,
I'm trying to use your library to deserialize a RecordBatch, could you add an example for that?

Thank you!

[Swoorup]

@slinkydeveloper

You can't directly deserialize from a RecordBatch, you'll have to convert it to a StructArray first using StructArray::from(record_batch)

[slinkydeveloper]

Tried this:

#[derive(Debug, Clone, PartialEq, ArrowField, ArrowSerialize, ArrowDeserialize)]
pub struct QueryResult {
    #[arrow_field(type = "arrow_convert::field::LargeString")]
    key: String,
    #[arrow_field(type = "arrow_convert::field::LargeBinary")]
    value: Vec<u8>,
}

arrow_array_deserialize_iterator::<QueryResult>(&StructArray::from(batch)).unwrap()

But got:

called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Data type mismatch")

Would be nice to have more info in this error :)

[slinkydeveloper]

Interesting, for my query i managed to get it working by making all the fields optional.

#[derive(Debug, Clone, PartialEq, ArrowField, ArrowSerialize, ArrowDeserialize)]
pub struct QueryResult {
    key: Option<String>,
    value: Option<Vec<u8>>,
}

Perhaps the problem is that if in my schema a field is nullable, then the field must be optional?

[Swoorup]

You probably have the Nullable flag enabled in one or more of your incoming data type.

Would certainly be nice to have more in-depth error reporting.

[slinkydeveloper]

Would certainly be nice to have more in-depth error reporting.

Will try to PR a little change to that error message :)

[slinkydeveloper]

See #4

[Swoorup]

Published 0.6.6