GH-45167: [C++] Implement Compute Equals for List Types #45272

WillAyd · 2025-01-15T16:38:51Z

Rationale for this change

While equality exists for ListScalars, it is not available through the compute module. This makes that now possible.

What changes are included in this PR?

I have added equals and not_equals support to the compute module for list types

Are these changes tested?

Yes - see added changes

Are there any user-facing changes?

Yes - the new feature to allow list comparison through the compute module

GitHub Issue: [C++][Python] Implement pc.equal for List arguments #45167

github-actions · 2025-01-15T16:39:19Z

⚠️ GitHub issue #45167 has been automatically assigned in GitHub to PR creator.

cpp/src/arrow/compute/kernels/codegen_internal.h

WillAyd · 2025-01-21T20:46:33Z

cpp/src/arrow/compute/kernels/codegen_internal.h

+  explicit ArrayIterator(const ArraySpan& arr) : arr(arr), position(0) {}
+
+  T operator()() {
+    const auto array_ptr = arr.ToArray();


The alternative to calling ToArray with the cast would be to implement something like value_slice on the ArraySpan directly, although I'm not sure if the ArraySpan is supposed to return anything but pointers to primitives (as is currently implemeted)

This is going to be slow, so we probably want to avoid this IMHO.

You may want to run a crude benchmark from Python to check this.

How about:

Get offset, length

Subslice the value array

Build ListScalar / LargeListScalar from the child array?

Or materialize the child array, and using the sub array. arr.ToArray() every call is too expansive?

WillAyd · 2025-01-23T14:15:47Z

@pitrou @jorisvandenbossche would either of you be able to take a look here?

pitrou · 2025-01-23T17:47:38Z

cpp/src/arrow/compute/kernels/scalar_compare.cc

@@ -445,6 +445,14 @@ std::shared_ptr<ScalarFunction> MakeCompareFunction(std::string name, FunctionDo
    DCHECK_OK(func->AddKernel({ty, ty}, boolean(), std::move(exec)));
  }

+  if constexpr (std::is_same_v<Op, Equal> || std::is_same_v<Op, NotEqual>) {
+    for (const auto id : {Type::LIST, Type::LARGE_LIST}) {
+      auto exec = GenerateList<applicator::ScalarBinaryEqualTypes, BooleanType, Op>(id);


Another approach with perhaps a better performance potential would be to leverage the existing RangeDataEqualsImpl in arrow/compare.cc.

Thanks for the heads up - I will give that a look. So I see all of the functions right now in the compare module are registered via RegisterScalarComparison. With what you are suggesting, I'm guessing I should be creating a new registry function along with that like RegisterRangeComparison right?

Also I'm guessing the RangeDataEqualsImpl is supposed to work when comparing two arrays, but not when comparing an array with a scalar

FWIW though I did benchmark the current implementation and it was definitely slow. Seemed about 1000x slower than an equivalent comparison using primitive types

With what you are suggesting, I'm guessing I should be creating a new registry function along with that like RegisterRangeComparison right?

I think we can avoid that by directly calling into RangeDataEqualsImpl.

Also I'm guessing the RangeDataEqualsImpl is supposed to work when comparing two arrays, but not when comparing an array with a scalar

A list scalar's value is actually an array, so that should not necessarily be a problem.

OK took a closer look at this. So AFAICT the RangeDataEqualsImpl returns a scalar bool value, rather than an array of booleans like we would need in the result here. That class is also private to the compare.cc module and doesn't expose any suitable entrypoint in compare.h that I think would work here.

Are you thinking we should refactor the RangeDataEqualsImpl to support vector functions and move it to make it accessible to the compute module, or do you think we should just create a dedicated class drawing some inspiration from it in scalar_compute.cc?

Thanks again for the guidance and patience here! Trying to wrap my head around the structure of the compute modules

So AFAICT the RangeDataEqualsImpl returns a scalar bool value, rather than an array of booleans like we would need in the result here.

That's right, so it would need to be called once for each list element (which is admittedly non optimal, but probably better than using GetScalar anyway?).

That class is also private to the compare.cc module and doesn't expose any suitable entrypoint in compare.h that I think would work here.

Well, we could add a suitable entrypoint in compare_internal.h if that's useful.

Another possible approach would be to leverage the comparison kernel for the child type, but that would probably be even more involved. So that's up to how much work you want to put into this :)

jorisvandenbossche · 2025-01-28T07:13:24Z

I currently have no time to review this in depth, but API-wise one remark: right now (for primitive arrays), nulls propagate in an operation like equal.
So how do they behave for nested typed, i.e. what if there is a null in a list element. Does that propagate as well (and does it make the comparison of the full list element null), or do we then consider a list element with nulls in the same location as equal?

WillAyd · 2025-01-28T14:42:55Z

The current (rather slow) implementation just does an elementwise compare, dispatching to the logical list scalar type. Therefore, since:

>>> l1 = pa.scalar([], type=pa.list_(pa.int32()))
>>> l2 = pa.scalar([], type=pa.list_(pa.int32()))
>>> l1 == l2
True

Wrapping that in an array does not change the behavior:

>>> arr1 = pa.array([l1])
>>> arr2 = pa.array([l2])
>>> pc.equal(arr1, arr2)
<pyarrow.lib.BooleanArray object at 0x71e8e9232560>
[
  true
]

mapleFU · 2025-02-06T09:36:43Z

cpp/src/arrow/compute/kernels/codegen_internal.h

@@ -390,6 +417,12 @@ struct UnboxScalar<Type, enable_if_has_string_view<Type>> {
  }
 };

+template <typename Type>
+struct UnboxScalar<Type, enable_if_list_type<Type>> {


So fixed_size_list is also declared as scalar, but not being registered in compute? ( It's ok to me, just to make sure this)

mapleFU · 2025-02-06T09:42:39Z

cpp/src/arrow/compute/kernels/codegen_internal.h

+  explicit ArrayIterator(const ArraySpan& arr) : arr(arr), position(0) {}
+
+  T operator()() {
+    const auto array_ptr = arr.ToArray();


How about:

Get offset, length

Subslice the value array

Build ListScalar / LargeListScalar from the child array?

Or materialize the child array, and using the sub array. arr.ToArray() every call is too expansive?

github-actions bot added Component: C++ awaiting review Awaiting review labels Jan 15, 2025

WillAyd commented Jan 15, 2025

View reviewed changes

cpp/src/arrow/compute/kernels/codegen_internal.h Outdated Show resolved Hide resolved

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jan 15, 2025

WillAyd force-pushed the implement-list-compare branch from 573867e to 649e3b0 Compare January 16, 2025 20:09

WillAyd commented Jan 17, 2025

View reviewed changes

cpp/src/arrow/compute/kernels/codegen_internal.h Outdated Show resolved Hide resolved

WillAyd force-pushed the implement-list-compare branch from 649e3b0 to 68bb513 Compare January 21, 2025 20:43

WillAyd marked this pull request as ready for review January 21, 2025 20:44

WillAyd commented Jan 21, 2025

View reviewed changes

pitrou reviewed Jan 23, 2025

View reviewed changes

WillAyd force-pushed the implement-list-compare branch from 68bb513 to 3b055a8 Compare January 27, 2025 22:47

apacheGH-45167: [C++] Implement Compute Equals for List Types

ba3b114

WillAyd force-pushed the implement-list-compare branch from 3b055a8 to ba3b114 Compare February 4, 2025 19:43

mapleFU reviewed Feb 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-45167: [C++] Implement Compute Equals for List Types #45272

GH-45167: [C++] Implement Compute Equals for List Types #45272

WillAyd commented Jan 15, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Jan 15, 2025

WillAyd Jan 21, 2025

pitrou Jan 23, 2025

mapleFU Feb 6, 2025

WillAyd commented Jan 23, 2025

pitrou Jan 23, 2025

WillAyd Jan 27, 2025

WillAyd Jan 28, 2025

pitrou Jan 29, 2025

WillAyd Feb 5, 2025

WillAyd Feb 5, 2025

pitrou Feb 6, 2025

jorisvandenbossche commented Jan 28, 2025

WillAyd commented Jan 28, 2025

mapleFU Feb 6, 2025

mapleFU Feb 6, 2025

GH-45167: [C++] Implement Compute Equals for List Types #45272

Are you sure you want to change the base?

GH-45167: [C++] Implement Compute Equals for List Types #45272

Conversation

WillAyd commented Jan 15, 2025 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Jan 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Jan 23, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Jan 28, 2025

WillAyd commented Jan 28, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Jan 15, 2025 •

edited by github-actions bot

Loading