GH-44248: [FORMAT] Add TimestampWithOffset canonical extension type

[serramatutu]

THIS IS A DRAFT. It's being used as reference for the [DISCUSS] thread in the mailing list.

Rationale for this change

Closes #44248

Arrow has no built-in canonical way of representing the TIMESTAMP WITH TIME ZONE SQL type, which is present across multiple different database systems. Not having a native way to represent this forces users to either convert to UTC and drop the time zone, which may have correctness implications, or use bespoke workarounds. A new arrow.timestamp_with_offset extension type would introduce a standard canonical way of representing that information.

Rust implementation: apache/arrow-rs#8743
Go implementation: apache/arrow-go#558

What changes are included in this PR?

Proposal and documentation for arrow.timestamp_with_offset canonical extension type.

Are these changes tested?

N/A

Are there any user-facing changes?

Yes, this is an extension to the arrow format.

• GitHub Issue: [Format] Support an official "timestamp with time zone offset" type #44248

[serramatutu]

Hey friends, I pushed 2 new patches, as discussed in the Arrow dev community meeting:

1. Made RFC3339 encoding to/from JSON a recommendation but not a requirement, and added a rationale on why this is recommended. Implementations can still provide ways to encode it as JSON objects without violating the spec if they want, though.

2. The offset_minutes field now can also be run-end or dictionary encoded for columns with lots of repeating offsets (as suggested in the mailing list).

Next, I need to change the Go and Rust implementation drafts to accept the new supported encodings.

[rok]

FWIW, forgot to mention this before in arrow-cpp we have an internal OffsetZone that represents offset in minutes. We define it per array, not per row of course.

arrow/cpp/src/arrow/util/date_internal.h

Lines 29 to 33 in cd23a76

	class OffsetZone {
	std::chrono::minutes offset_;
	public:
	explicit OffsetZone(std::chrono::minutes offset) : offset_{offset} {}

[pitrou]

This is not an extension type parameter, it's a parameter of the storage type.

[serramatutu]

Resolved: 043b07c

[pitrou]

This makes consumption of these data more complicated. Why not mandate that struct fields are always non-null?

[felipecrv]

And return an error on instantiation?

I think we can allow nulls on the timestamp field and on instantiation put it on the struct array also.

Would you be fine with both points above?

[felipecrv]

I think many operations will care only about the timestamp field and having the validity buffer there helps them simply delegate to functions on the Timestamp type. At the same time, many function only look at the top-level validity buffer (i.e. the validity of the struct). So during instantiation it would make sense to share the validity buffer if there is one.

[pitrou]

And return an error on instantiation?

Which instantiation are we talking about exactly? The ExtensionArray subclass in Arrow C++?

I think we can allow nulls on the timestamp field and on instantiation put it on the struct array also.

Would you be fine with both points above?

We can, but that places a burden on the consumer for the sake of making things easier for the producer. I'm not sure that's a good tradeoff to make.

I think many operations will care only about the timestamp field and having the validity buffer there helps them simply delegate to functions on the Timestamp type. At the same time, many function only look at the top-level validity buffer (i.e. the validity of the struct). So during instantiation it would make sense to share the validity buffer if there is one.

I admit that can make sense. Let's hope for other opinions on the topic. @paleolimbot @alamb @tustvold

[felipecrv]

And return an error on instantiation?

Which instantiation are we talking about exactly? The ExtensionArray subclass in Arrow C++?

In Arrow implementations for any language.

[felipecrv]

Using the validity buffer of the timestamp field for both the struct and the field itself is good for both producers and consumers. A producer doesn't have to set default values for the nulls. And consumers don't have to look at both validity buffers for anything they do. Sharing the buffer (if it exists at all) is just a ref-count bump on the buffer.

[paleolimbot]

Why not mandate that struct fields are always non-null

+1. I had the same worry when we decided to make all of GeoArrow's native memory layouts have non-nullable children...the language in GeoArrow is that children must not contain nulls. When producing, all the implementations explicitly mark fields as non-nullable. When consuming, they check for a zero null count. Specifying anything about the specific arrangement of buffers I think is out of scope for an extension type definition (if we want to do this, somebody has to make sure it can be done in places like DuckDB, JavaScript, and Julia that might not offer the same level of control over buffer creation).

[pitrou]

Well, if that's the intent, the spec needs to be clear that all 3 validity buffers have to be the identical.

But that would be frankly a bizarre variation from normal Struct policies and I don't see any reason to constrain implementations in such a way.

[felipecrv]

Good. We are actually back to the very original proposal by @serramatutu that demanded non-nullable fields.

[serramatutu]

Latest change goes back to the original spec: 043b07c

[github-actions]

⚠️ GitHub issue #44248 has been automatically assigned in GitHub to PR creator.

[alamb]

Do we want to say anything about what types of dictionary keys are allowed? Like is it ok to use Int8/Int16/Int32 keys -- while Int32 probably doesn't make sense as it would make the column larger, it might be necessary in some cases to encode the full range of Int16 values 🤔

[felipecrv]

In the interest of brevity of the spec, I advised @serramatutu to not have recommendations here. I would assume anyone deciding to dictionary-encode an int16 array is considering the gains (or not) that they would have from that encoding. We are allowing but not recommending complicated encoding of an already compact 16-bit-valued array.

[alamb]

Thanks for this PR @serramatutu -- the basic idea looks good to me