GH-45159: [CI][Integration] Remove substrait consumer-testing integration job

[raulcd]

Rationale for this change

The job has been failing for the last two months due to upstream refactoring. I opened an issue upstream but we didn't got any response:

• Apache Arrow integration with substrait consumer-testing job is failing and not maintained - Removal proposal substrait-io/consumer-testing#196

Based on the following substrait consumer-testing PR there was a big refactor which removed the files that we were invoking on our tests:

• feat: factor out result generation and remove old test_*.py substrait-io/consumer-testing#162

I asked on the corresponding issue ~3 weeks ago for feedback on the removal and as there's no opposition I think we should remove this unmaintained job.

What changes are included in this PR?

Remove extremely coupled substrait-io/consumer-testing job.

Are these changes tested?

No

Are there any user-facing changes?

No

• GitHub Issue: [Python][CI] Substrait nightly job is failing due to wrong invocation of substrait consumer-testing #45159

[github-actions]

⚠️ GitHub issue #45159 has been automatically assigned in GitHub to PR creator.

[ingomueller-net]

LGTM!

Thanks for letting us know. I wasn't aware that there were any downstream users when I did the refactoring.

In the mid or long term, what would be requirements/wishes for a Substrait test suite for Arrow to consume?

[raulcd]

Hi @ingomueller-net thanks for answering. From my understanding this was testing possible regressions for pyarrow-dev acero's consumer on the previous test_boolean_functions.py mainly on our nightly jobs. I am unsure whether more tests could be run or not. There is value on validating that new developments continue passing the integration validation. The problem arises on the lack of expertise on the consumer-testing on our repository.

[ingomueller-net]

OK, thanks for the input. To paraphrase and add my own interpretation: I imagine you would like to have a suite of Substrait plans that you can run against the Arrow consumer as end-to-end tests that tell you whether the consumer respects the spec, right? Ideally, it would allow you to skip tests that you know the consumer currently (1) doesn't support or (2) produces wrong results (such that you can, at least, ensure that there are no regression on the others). You might even want the test suite to contain invalid Substrait plans that allow you to test whether your consumer correctly rejects them. Is that so?

[raulcd]

That sounds reasonable to me but I am unsure on what is the current substrait status on Arrow. Probably @westonpace or @amol- know more about it.