[VL][Delta] Add native Delta DV reader support by malinjawi · Pull Request #12040 · apache/gluten

malinjawi · 2026-05-05T15:20:53Z

What changes are proposed in this pull request?

This PR is the second step in the split Delta deletion-vector (DV) stack, following #12001.

It adds the native Velox-side Delta DV reader layer that consumes the roaring bitmap payload facilities introduced by #12001, without adding the JVM-side Delta scan metadata handoff yet.

Main changes:

add a native Delta connector and data source backed by the Hive connector/data source infrastructure
register a scoped Delta connector alongside the existing scoped Hive connector for each Velox runtime
add Delta split metadata types for:
- deletion-vector descriptors
- protocol metadata
- file statistics used for DV validation
- serialized split payload buffer views
add DeltaDeletionVectorReader to load materialized Delta DV payloads using RoaringBitmapArray
add DeltaSplitReader to validate DV protocol/statistics metadata and apply row-index filtering semantics
add focused native unit coverage for connector setup, split metadata, and deletion-vector reader behavior

This PR is intentionally native-reader only:

no JVM-side Delta scan metadata handoff yet
no end-to-end Delta scan offload behavior change yet

Those pieces will be added in follow-up split PRs.

issue #11901.

How was this patch tested?

Added focused native test coverage in:

cpp/velox/compute/delta/tests/DeltaConnectorTest.cpp
cpp/velox/compute/delta/tests/DeltaSplitTest.cpp
cpp/velox/compute/delta/tests/DeltaDeletionVectorReaderTest.cpp

Covered cases:

Delta connector configuration and connector properties
split-carried deletion-vector descriptors and logical row-count accounting
loading materialized DV payloads from RoaringBitmapArray
row deletion checks and keep/drop filter decisions
empty payload handling and invalid payload rejection
protocol/statistics validation for DV-bearing splits

Validation run:

fork preview CI against malinjawi/incubator-gluten:main on the combined PR2 branch: all checks passed after rerunning two infra-flaky jobs
local git diff --check upstream/main...HEAD
local clang-format pass with /opt/homebrew/opt/llvm@15/bin/clang-format over changed C++ files

Was this patch authored or co-authored using generative AI tooling?

Generated-by: IBM BOB

github-actions · 2026-05-06T11:08:46Z

Run Gluten Clickhouse CI on x86

zhztheplayer

Thanks.

zhztheplayer · 2026-05-13T14:29:47Z

+struct DeltaProtocolInfo {
+  int32_t minReaderVersion;
+  int32_t minWriterVersion;
+  std::optional<std::vector<std::string>> readerFeatures;
+  std::optional<std::vector<std::string>> writerFeatures;
+
+  /// Check if this protocol supports deletion vectors.
+  /// Returns true if:
+  /// - minReaderVersion >= 3
+  /// - minWriterVersion >= 7
+  /// - 'deletionVectors' is in readerFeatures
+  bool supportsDeletionVectors() const {
+    if (minReaderVersion < 3 || minWriterVersion < 7) {
+      return false;
+    }
+    if (!readerFeatures.has_value()) {
+      return false;
+    }
+    return std::find(readerFeatures->begin(), readerFeatures->end(), "deletionVectors") != readerFeatures->end();
+  }
+};


In which case do we need this struct and the validation logics? Won't we only use native DV features when the Java side feature exists?

Good point @zhztheplayer, this was carried over from the earlier delta-dv-read-foundation branch, where native had a broader/future-scoped DV descriptor path.

For this split PR, native only consumes JVM-materialized DV payloads. The Java-side Delta integration is responsible for gating DV support before handing the payload to native, so this protocol struct and validation path are not needed in the current scope.

I removed DeltaProtocolInfo and the related validation logic from this PR. Native now keeps only the checks it actually needs here: materialized payload presence, optional cardinality validation, and file statistics consistency.

I will take a note of that and maybe we can reintroduce protocol validation later if/when native directly resolves Delta DV descriptors.

zhztheplayer · 2026-05-13T14:33:13Z

+
+namespace {
+
+class DeltaConnectorTest : public ::testing::Test {


Can we also add some test cases that are more E2E? E.g., read a file with a passed-in DV, then verify whether the unwanted rows are filtered out?

Good catch @zhztheplayer

Added a native test for this:
https://github.com/apache/gluten/pull/12040/changes#diff-c7e9378ffacac4c918121006a2b4606297a5a321fa34ab432e82ef9748aead4bR148-R167

The test writes a small DWRF file, scans it through the Delta connector with a materialized DV payload passed in the split, and verifies that rows whose file positions are in the DV are filtered out from the scan result.

zhztheplayer · 2026-05-13T14:37:22Z

+  void loadSerializedDeletionVector(
+      std::string_view serializedPayload,
+      std::optional<uint64_t> expectedCardinality = std::nullopt);


Can we add comments explaining expectedCardinality?

Thanks @zhztheplayer!

Updated the comment to explain expectedCardinality more directly:

https://github.com/apache/gluten/pull/12040/changes/BASE..e2aeaed2de9e03de82f61968faa83c51c5bf7c7e#diff-31100294214221e6cd8528fb9ae69506a6a3d6e45e4f24b126146b8db3d1863cR67-R76

github-actions Bot added VELOX CORE works for Gluten Core labels May 5, 2026

Mohammad Linjawi added 2 commits May 7, 2026 12:28

[VL][Delta] Add native Delta DV reader support

48b59e5

fix: define split payload view for native Delta reader

66ea460

malinjawi force-pushed the split/delta-dv-native-reader-pr branch from 1a16894 to 66ea460 Compare May 7, 2026 09:29

github-actions Bot removed the CORE works for Gluten Core label May 7, 2026

fix: align Delta reader with split IO stats API

e2aeaed

malinjawi mentioned this pull request May 13, 2026

[VL][Delta] Add JVM-side Delta DV scan handoff malinjawi/incubator-gluten#3

Draft

zhztheplayer reviewed May 13, 2026

View reviewed changes

Mohammad Linjawi added 2 commits May 14, 2026 13:19

fix: address Delta DV native reader review

905e6c8

fix: initialize folly in Delta read test

6be05d2

malinjawi force-pushed the split/delta-dv-native-reader-pr branch from 08353bc to 6be05d2 Compare May 14, 2026 12:32

docs: clarify Delta DV expected cardinality

4843cb8

malinjawi force-pushed the split/delta-dv-native-reader-pr branch from 0338c08 to 4843cb8 Compare May 14, 2026 19:48

Merge branch 'main' into split/delta-dv-native-reader-pr

82619e5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VL][Delta] Add native Delta DV reader support#12040

[VL][Delta] Add native Delta DV reader support#12040
malinjawi wants to merge 7 commits into
apache:mainfrom
malinjawi:split/delta-dv-native-reader-pr

malinjawi commented May 5, 2026

Uh oh!

github-actions Bot commented May 6, 2026

Uh oh!

zhztheplayer left a comment

Uh oh!

zhztheplayer May 13, 2026

Uh oh!

malinjawi May 14, 2026

Uh oh!

zhztheplayer May 13, 2026

Uh oh!

malinjawi May 14, 2026

Uh oh!

zhztheplayer May 13, 2026

Uh oh!

malinjawi May 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		namespace {

		class DeltaConnectorTest : public ::testing::Test {

Conversation

malinjawi commented May 5, 2026

Uh oh!

github-actions Bot commented May 6, 2026

Uh oh!

zhztheplayer left a comment

Choose a reason for hiding this comment

Uh oh!

zhztheplayer May 13, 2026

Choose a reason for hiding this comment

Uh oh!

malinjawi May 14, 2026

Choose a reason for hiding this comment

Uh oh!

zhztheplayer May 13, 2026

Choose a reason for hiding this comment

Uh oh!

malinjawi May 14, 2026

Choose a reason for hiding this comment

Uh oh!

zhztheplayer May 13, 2026

Choose a reason for hiding this comment

Uh oh!

malinjawi May 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants