perf: prune ListingTable files before grouping

[rgbuilds]

Which issue does this PR close?

Closes #26130.

Rationale for this change

When files contain disjoint ranges of a filtered column, ListingTable groups all files before file-statistics pruning. The matching files can therefore land in a single scan partition, and scan statistics still include files that cannot match.

In the eight-file example from the issue, d >= 20 AND d < 40 currently places both matching files in the second of four groups and estimates 80 rows. Pruning before grouping retains only f2 and f3, puts them in separate groups, and aggregates 20 survivor rows.

What changes are included in this PR?

• Pass non-partition filters through a private listing path and reuse PruningPredicateBuilder with already collected file statistics before splitting regular scan file groups.
• Keep files with missing/inexact statistics or evaluation errors, retain normal row filtering, and avoid evaluating volatile filters during planning.
• Bypass the optimization for custom expression adapters and declared output partitioning, preserving partition identities and empty trailing groups.
• Recompute aggregate statistics from survivors without additional object-store requests.
• Add regression coverage and update physical-plan/runtime-metric expectations for files eliminated during planning. No public API change.

What is the testing strategy for this PR?

On the investigated baseline (a4b8198cf), local validation passed 12,579 distinct Rust tests and 533 SQL logic files. This includes 23 listing unit tests, 10 focused pruning integration tests, all 246 Parquet integration tests, and 126 extended fuzz tests. The no-extra-I/O test compares object-store requests during planning.

Coverage includes the eight-file grouping example, partially matching files, missing and inexact bounds, NULL semantics, Hash/Range partition layouts, Hive/data predicates, LIMIT/OFFSET, custom adapters, volatile UDF invocation counts, survivor statistics, and conservative filtered-statistics precision. Nine SQL snapshot files change physical-plan expectations only.

On current upstream main (5f2158269), all 1,459 core integration tests (including all 10 pruning regressions and the no-extra-I/O test), all 247 Parquet tests, and all 535 SQL logic files pass. The full extended workspace run completed successfully: 12,742 Rust tests passed, including all 126 fuzz tests, with eight tests ignored and no failures. All 535 SQL logic files also passed. Formatting, diff whitespace, TOML formatting, and license headers on all 2,331 tracked project source files pass.

The earlier full workspace Clippy run passed with explicit allowances for pre-existing upstream/toolchain diagnostics. The mandatory strict Clippy invocation also fails on the updated main branch with Rust 1.99, on three pre-existing clippy::mutable_key_type diagnostics in untouched datafusion/optimizer/src/push_down_filter.rs. The CI-style Clippy invocation (--workspace --all-targets --features avro,integration-tests,extended_tests -- -D warnings) passes. The full lint wrapper stops at its license walk because locally exported third-party SQLite fixture snapshots lack project headers; the separate tracked-project-source license check passes. Subsequent wrapper steps have not been run. No lint suppression or unrelated optimizer change is included in this PR. No performance speedup is claimed: the prior Hive-only planning benchmark is not representative of data-column pruning and ran under differing concurrent load.

Are there any user-facing changes?

With statistics collection enabled, regular listing scans can contain fewer files and report tighter statistics in EXPLAIN. Query results and row filtering are unchanged. Unsupported or inconclusive statistics keep files, and declared partition layouts are preserved.

This contribution was developed with AI assistance. The design and safeguards are described above; performance under distributed execution remains unmeasured.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.75776% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.90%. Comparing base (5f21582) to head (bebaf2e).

Files with missing lines	Patch %	Lines
datafusion/catalog-listing/src/table.rs	98.75%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #26441    +/-   ##
========================================
  Coverage   82.89%   82.90%            
========================================
  Files        1154     1154            
  Lines      457620   457778   +158     
  Branches   457620   457778   +158     
========================================
+ Hits       379352   379520   +168     
+ Misses      55262    55249    -13     
- Partials    23006    23009     +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rgbuilds]

@andygrove This is ready for review and addresses #26130 by pruning files before grouping, using already collected statistics. Would appreciate your feedback when you have time.