ENH: first-class UUID support

[flying-sheep]

Feature Type

• Adding new functionality to pandas
• Changing existing functionality in pandas
• Removing existing functionality in pandas

Problem Description

A lot of people want to use UUID arrays, so we should provide first-class support.

If you want to play around with it, I prototyped this functionality here: https://pypi.org/project/pandas-uuid/

Feature Description

I suggest to do something similar as for {Arrow,}StringArray, i.e. have a dyad of ExtensionArray types, one backed by numpy and one by pyarrow.

They don’t need a lot of features except for comparison (self == other / self == elem) and membership tests (self.__contains__(elem)/elem in self and self.isin(other)).

The numpy variant needs to be backed by a np.void(16)/"V16" array, since np.bytes_/"S" has special treatment of null bytes, whereas numeric types (like UUIDs) assign no special meaning to them.

Alternative Solutions

• We could use MaskedArray as base for the numpy-backed variant to allow missing values in both cases.
• We could just force people to rely on pyarrow for this functionality, but I feel that wrapping np.void(16) is simple enough. People didn’t like adding the pyarrow dependency for basic functionality, so I assume you want to keep adding basic features that don’t rely on pyarrow.
• We could keep the pandas-uuid package around, see here for its limitations.

Additional Context

• Turns out the following issue isn’t actually a blocker if we have the ExtensionDtype report its kind to be "O": BUG: Series constructor incorrectly assumes that any dtype with .kind == 'V...' is a “compound dtype” #54810
• We should test that things like this work by default: BUG: DataFrame to JSON failed when it with UUID #59132

[rhshadrach]

Thanks for the request - as another alternative for pyarrow, you can use ArrowExtensionArray.

s = pd.Series([uuid4().bytes], dtype=pd.ArrowDtype(pa.uuid()))
print(s)
# 0    b'\xee\xfb+\xd0\x0bOKd\x85d\xa1\xd6\xf4\x1eY\x1e'
# dtype: extension<arrow.uuid>[pyarrow]

It's not clear to me what benefit a custom EA has over this.

[flying-sheep]

See “Alternative Solutions” point 2.

[rhshadrach]

Perhaps I'm misunderstanding - you're proposing to add an EA specifically for PyArrow-backed UUID data, is that right? It's not clear what advantage that would have over the current ArrowExtensionArray.

The NumPy case is clear.

[flying-sheep]

Ah got it! You’re basically suggesting “Alternative Solution 2b”: Only add the numpy EA but keep the unspecific pyarrow one.

Wrapping the pyarrow case has smaller benefits. Most importantly, the behavior of the two storage backends could be more transparent, and people could just create a dtype="uuid" and not care about behavioral differences, just like they can now with dtype="string" arrays.

Some specific advantages:

• (un)boxing has many convenience benefits, like allowing if my_uuid in pyarrow_backed: and so on instead of manually converting objects to pyarrow.UuidScalar (e.g. in your code s[0] in s evaluates to False)
• element-wise comparisons between the numpy version and pyarrow version would work (maybe partially possible by implementing everything on the numpy EA side?)
• currently doing pd.Series(pa.array(..., type=pa.uuid())) creates a series with object dtype instead of the extension dtype
• We should display UUIDs using the canonical UUID representation, as opposed to the output of your example

Some advantages might be obtainable by special-casing some code paths in ArrowExtensionArray, like we do for timestamps, but it’s unclear to me why we chose to do that for timestamps while having a subclass for strings, and where to draw the line.

[flying-sheep]

What’s there in pandas is less usable than I thought, I couldn’t figure out at all how to make __contains__ work without accessing _pa_array directly:

So s[0] conveniently unboxes to a uuid.UUID object, but then?

>>> s[0] in s
False  # Should be True!
>>> pa.scalar(s[0].bytes, type=pa.uuid()) in s
False  # Should be True?
>>> s[0] in s.array
NotImplementedError: extension<arrow.uuid>
>>> pa.scalar(s[0].bytes, type=pa.uuid()) in s.array
NotImplementedError: eq not implemented for <class 'pyarrow.lib.UuidScalar'>
>>> pa.scalar(s[0].bytes, type=pa.uuid()) in s.array._pa_array
True  # but at what cost?

All this might be fixable generically (maybe fixing the NotImplementedErrors makes the rest work), but point is, none of does yet, and with some APIs simply returning the wrong result instead of erroring, it’s not super reassuring to use.