…or partition pruning (apache#12092)

* [GLUTEN-XXXXX][VL] PA-1 RED: ColumnarCachedBatchKryoSuite for stats field + magic-prefix sniff

PA-1 RED tests for Layer A min/max stats — pure JVM-only suite, no native lib.

3 RED cases (Expected RED failure column see plan §3 PA-1):
- PA-1.1 testStatsFieldRoundTripV2: case class only has 3 fields → compile error
  'too many arguments' until stats: InternalRow added
- PA-1.2 testKryoV1Backwards: v1 binary read should yield stats=null (graceful
  degrade) → fails until v2 reader handles v1 layout
- PA-1.3 testV1BinaryNumRowsHigh02NotMisidentified: NB1 ship-blocker — v1 with
  numRows=258 (first byte 0x02) must NOT be mis-identified as v2 by naive
  first-byte sniff → magic-prefix 4-byte sniff (0xFECA5302) is the only correct
  approach

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md PA-1
refs: todos/features/gluten-inmemory-cache-stats/docs/0002-cpp-stats-contract.md §4
refs: todos/features/gluten-inmemory-cache-stats/docs/0003-jni-binary-framing-reference.md §9

* [GLUTEN-XXXXX][VL] PA-1 GREEN: CachedColumnarBatch stats field + Kryo magic-prefix sniff (BE)

GREEN impl for PA-1 RED tests (ColumnarCachedBatchKryoSuite). PA-1 ONLY adds
the stats field + Kryo serializer; buildFilter override + extends
SimpleMetricsCachedBatchSerializer is deferred to PA-3 per plan 0004.

(1) CachedColumnarBatch case class:
  - Add 4th field 'stats: InternalRow' (nullable; null = stats unavailable,
    triggers buildFilter pass-through fallback in graceful degrade)
  - Change base from CachedBatch to SimpleMetricsCachedBatch (SPARK-32274 trait,
    dormant until PA-3 switches serializer parent)
  - sizeInBytes scaladoc clarifies Gluten override semantic (serialized blob
    length, not vanilla per-column-sum)

(2) CachedColumnarBatchKryoSerializer:
  - V2_MAGIC = 0xFE 0xCA 0x53 0x02 (4-byte sniff prefix)
    Cannot collide with v1 binary because v1 first 4 bytes are numRows in
    Kryo fixed 4-byte BIG-ENDIAN; any non-negative Int < 2^31 has BE high
    byte in [0x00, 0x7F]. Magic high byte 0xFE > 0x7F is structurally
    unreachable, eliminating all single-byte aliasing vectors.
  - write path: emit V2_MAGIC + numRows (Kryo Output.writeInt(int) single-arg
    = fixed 4-byte BE; AVOID (int, boolean) overload which forwards to
    writeVarInt 1-5 bytes and breaks the magic sniff invariant) + sizeInBytes
    + bytes + hasStats Boolean + optional (statsLen + statsBytes)
  - read path: peek 4-byte first; if matches V2_MAGIC -> readV2; else ->
    readV1 (treats first4 as raw numRows BE, returns stats=null for
    graceful degrade)
  - Companion object holds V2_MAGIC + serializeStats / deserializeStats helpers
    Java serialization placeholder; PA-3 MUST replace with statsBlob binary
    framing per contract 0002 §3 + reference 0003 §3-4 before any code path
    can produce stats != null (Java ser is NOT cross-version checkpoint safe).

(3) convertColumnarBatchToCachedBatch L347-349 write path:
  - Add stats=null placeholder. PA-3 replaces with stats InternalRow built
    from cpp computeStats output via JNI serializeWithStats.

NOTE: still extends bare CachedBatchSerializer (not SimpleMetricsCachedBatchSerializer
yet); buildFilter pass-through L426-431 unchanged. PA-3 will switch base + delete
pass-through + add SimpleMetricsCachedBatch invariant debug assert.

review: reviews/batch-PA-1-code-review.md (Layer 2 三 personas; 1 BLOCKER B1
Kryo wire format BE/LE 修正已 amend; verdicts P1 -1->+0 / P2 不通过->通过 / P3 +1)

refs: todos/features/gluten-inmemory-cache-stats/docs/0001-layerA-minmax-design.md D-A2
refs: todos/features/gluten-inmemory-cache-stats/docs/0002-cpp-stats-contract.md §4 (BL3 + NB1 + rev 3 BE)
refs: todos/features/gluten-inmemory-cache-stats/docs/0003-jni-binary-framing-reference.md §9 (rev 3 BE)
refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md PA-1
refs: todos/features/gluten-inmemory-cache-stats/reviews/batch-PA-1-code-review.md

* [GLUTEN-XXXXX][VL] PA-2.1 RED: VeloxColumnarBatchSerializer::computeStats(Long FlatVector)

Locks the minimal ColumnStats contract:
  struct ColumnStats { hasLowerBound, hasUpperBound, lowerBound, upperBound, nullCount }
  std::vector<ColumnStats> computeStats(RowVectorPtr)

RED stderr (gcc, --build_tests=ON):
  error: '...VeloxColumnarBatchSerializer' has no member named 'computeStats'
  (+4 cascade errors on stats[0].lowerBound.value<int64_t>())

PA-2 batch micro-sliced: PA-2.1 = Long FlatVector path only. PA-2.2-2.6
(NaN / nulls absent / Decimal / JNI framing / capability) follow each as
one-RED -> one-production-change increments.

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-2.1 GREEN: ColumnStats + computeStats(BIGINT FlatVector)

Adds ColumnStats struct and VeloxColumnarBatchSerializer::computeStats(rowVector)
scalar v1 -- BIGINT FlatVector min/max + nullCount via direct buffer scan.
Other type kinds fall back to hasLowerBound=hasUpperBound=false (graceful
degrade: downstream buildFilter override emits pass-through for that column).

GREEN log (gtest, --gtest_filter='*PA_2_1*'):
  [ RUN      ] VeloxColumnarBatchSerializerTest.PA_2_1_testComputeStatsLongFlatVector
  [       OK ] VeloxColumnarBatchSerializerTest.PA_2_1_testComputeStatsLongFlatVector (0 ms)
  [  PASSED  ] 1 test.

PA-2.2..2.6 (NaN / nulls absent / Decimal / JNI framing / capability) follow.

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-2.2 RED: NaN Float partition must not participate in pruning

RED stderr:
  Value of: stats[0].hasLowerBound
    Actual: false
    Expected: true
  REAL FlatVector w/o NaN must be supported after PA-2.2 GREEN

PA-2.1 sentinel: PASSED (regression OK).

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-2.2 GREEN: REAL FlatVector scan with NaN poison guard

Refactor computeStats to a templated scanMinMax<T>() helper covering BIGINT
and REAL. Floating-point specialization aborts on first NaN value -- per
Spark equality semantics (NaN != NaN), any NaN poisons the whole column so
min/max-based pruning would silently drop matching rows. NB4 ship blocker
invariant: NaN -> hasLowerBound=hasUpperBound=false -> buildFilter
pass-through.

GREEN log (*PA_2_*):
  PA_2_1_testComputeStatsLongFlatVector ........ OK
  PA_2_2_testComputeStatsNaNFloatPartition ..... OK
  2 PASSED.

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-2.4 RED: HUGEINT (long Decimal P>=19) FlatVector min/max

PA-2.3 (nulls-buffer-absent) folded into PA-5 ship blocker UT batch -- the
existing GREEN already has a defensive 'if (nulls != nullptr)' guard, so no
honest RED can be authored against an already-correct path (Plan sec 0
ironclad rule 2: no fake 'already green' RED).

RED stderr:
  Value of: stats[0].hasLowerBound  Actual: false  Expected: true
    HUGEINT (long Decimal P>=19) FlatVector must be supported after PA-2.4 GREEN
  Value of: stats[0].hasUpperBound  Actual: false  Expected: true
  C++ exception 'wrong kind! UNKNOWN != HUGEINT'
    -- raised by stats[0].lowerBound.value<int128_t>() because the default-
       constructed variant has KIND::UNKNOWN; future GREEN populates it.

PA-2.1/2.2 sentinels: PASSED.

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-2.4 GREEN: HUGEINT (long Decimal P>=19) FlatVector scan

Adds HUGEINT case to computeStats switch -- reuses scanMinMax<int128_t>
template (no NaN guard via if constexpr branch). variant(int128_t) ctor
exists in Velox so no special variant::create<HUGEINT> needed.

JVM-side 16-byte big-endian BigInteger marshaling deferred to PA-3 (writer
path) per Plan sec PA-3.

GREEN log (*PA_2_*):
  PA_2_1_testComputeStatsLongFlatVector ............... OK
  PA_2_2_testComputeStatsNaNFloatPartition ............ OK
  PA_2_4_testComputeStatsHugeintDecimalFlatVector ..... OK
  3 PASSED.

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-2.5a RED: framedSerializeWithStats top-level layout

Locks framing layout per docs/0003-jni-binary-framing-reference.md sec 2:
  [ magic(4) | statsLen(4 LE) | statsBlob | bytesLen(4 LE) | bytesBlob ]
where magic = 0xFE 0xCA 0x53 0x02 (matches PA-1 V2_MAGIC).

PA-2.5a scope: framing + bytesBlob (delegated to existing serializeTo) +
EMPTY statsBlob (statsLen=0). PA-2.5b will populate statsBlob; PA-2.5c
adds JNI bridge.

Note byte-order asymmetry: cpp framing headers are LE (0003 sec 2);
JVM-side Kryo wrap (PA-1) uses single-arg writeInt = BE. Different layers,
different conventions; both intentional, both documented.

RED stderr:
  error: '...VeloxColumnarBatchSerializer' has no member named 'framedSerializeWithStats'

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-2.5a GREEN: framedSerializeWithStats top-level layout

Adds framedSerializeWithStats(batch) returning std::vector<uint8_t> framed as
[ magic(4) | statsLen(4 LE) | statsBlob | bytesLen(4 LE) | bytesBlob ].
PA-2.5a uses an empty statsBlob (statsLen=0) -- bytesBlob delegated to
existing append+serializeTo path, so end-to-end round-trip via existing
deserialize remains intact (PA-3 will skip the framing header on read).

PA-2.5b will populate statsBlob from computeStats output; PA-2.5c adds the
JNI bridge entry point Java_..._serializeWithStats.

GREEN log (*PA_2_*):
  PA_2_1_testComputeStatsLongFlatVector ............... OK
  PA_2_2_testComputeStatsNaNFloatPartition ............ OK
  PA_2_4_testComputeStatsHugeintDecimalFlatVector ..... OK
  PA_2_5a_testFramedSerializeWithStatsLayout .......... OK
  4 PASSED.

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-2.5b RED: statsBlob layout (BIGINT 1-col)

Locks per-column stats marshaling per docs/0003-jni-binary-framing-reference
sec 3, BIGINT path only:
  [ numCols u32 LE ]
  per col: [ supported u8 ][ nullCount u32 LE ][ count u32 LE ][ sizeInBytes u64 LE ]
  if supported: [ lowerBoundLen u32 LE ][ lowerBound bytes ][ upperBoundLen u32 LE ][ upperBound bytes ]

PA-2.5b uses sizeInBytes=0 placeholder; PA-3 vanilla path may overwrite from
SimpleMetricsCachedBatch context (or stays 0 -- not consumed by partition
pruning). REAL/HUGEINT marshaling deferred to PA-2.5b-real / -hugeint
follow-up slices to keep one-RED-one-prod-change discipline (Plan sec 0
rule 1).

RED stderr:
  Expected: (statsLen) >= (4u), actual: 0 vs 4
    PA-2.5b: statsBlob must contain at least numCols(uint32)

(PA-2.5a writes statsLen=0; populating it is precisely the GREEN of PA-2.5b.)

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-2.5b GREEN: statsBlob BIGINT marshal + PA-2.5a sentinel relax

GREEN delta:
- framedSerializeWithStats now invokes computeStats(rowVector) and serializes
  per-column [supported u8 | nullCount u32 LE | count u32 LE | sizeInBytes u64 LE
  | (if supported) lowerBoundLen u32 LE | bytes | upperBoundLen u32 LE | bytes].
- BIGINT path emits 8-byte LE int64 for lower/upperBound; REAL/HUGEINT marshaling
  is held off behind 'emitSupported &&= variant.kind()==BIGINT' guard so we
  don't desync wire format with parsers expecting 8B (PA-2.5b-real / -hugeint
  follow-ups will widen this).
- sizeInBytes = 0 placeholder.

PA-2.5a sentinel relax: removed EXPECT_EQ(statsLen, 0u) -- it was a temporary
PA-2.5a-scope assumption mistakenly written as a permanent invariant. Lesson:
RED tests should only assert permanent contract shape, not transient placeholder
values from a prior micro-slice.

GREEN log (*PA_2_*):
  PA_2_1_testComputeStatsLongFlatVector ............... OK
  PA_2_2_testComputeStatsNaNFloatPartition ............ OK
  PA_2_4_testComputeStatsHugeintDecimalFlatVector ..... OK
  PA_2_5a_testFramedSerializeWithStatsLayout .......... OK (sentinel)
  PA_2_5b_testStatsBlobBigintLayout ................... OK
  5 PASSED.

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-2.5c: JNI bridge serializeWithStats (non-TDD plumbing)

Wires the JVM serializeWithStats(handle): byte[] native call to the cpp
framedSerializeWithStats virtual hook, end-to-end:

  cpp/core/operators/serializer/ColumnarBatchSerializer.h
    + virtual std::vector<uint8_t> framedSerializeWithStats(batch) -- default
      returns empty (other backends inherit the safe fallback).

  cpp/velox/operators/serializer/VeloxColumnarBatchSerializer.h
    + override marker on framedSerializeWithStats (impl already shipped in
      PA-2.5a/b commits).

  cpp/core/jni/JniWrapper.cc
    + Java_..._serializeWithStats: retrieve ColumnarBatch, call
      serializer->framedSerializeWithStats, convert to jbyteArray. Mirrors
      the existing _serialize entry point's structure.

  gluten-arrow/.../ColumnarBatchSerializerJniWrapper.java
    + native byte[] serializeWithStats(long handle).

Non-TDD pure plumbing batch (Plan sec 0 ironclad rule 2 explicitly allows
this category when test infra would dwarf the change). Verification:
  - cpp gtest sentinel: 5/5 PASS (PA-2.1/2.2/2.4/2.5a/2.5b unchanged).
  - libgluten.so exports symbol:
      T Java_org_apache_gluten_vectorized_ColumnarBatchSerializerJniWrapper_serializeWithStats
      W _ZN6gluten23ColumnarBatchSerializer24framedSerializeWithStats...
  - JVM mvn compile -pl gluten-arrow,backends-velox -am: BUILD SUCCESS (9/9).

End-to-end live invocation will be exercised by PA-3 (vanilla cache write
path calls jni.serializeWithStats) -- that is the natural integration test
for this plumbing.

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-2.6: capability check supportsStatsExt (non-TDD plumbing)

Adds ColumnarBatchSerializerJniWrapper.supportsStatsExt() static lazy
helper. PA-3 write path will branch on this -- true => jni.serializeWithStats,
false => legacy jni.serialize. Enables jar/native-lib version skew users
without UnsatisfiedLinkError surfacing.

Implementation note: probing via reflection on the declared native method
(presence => JVM side wired). The cpp side symbol is checked by the JNI
linker on first real invocation; PA-3 must catch UnsatisfiedLinkError there
too (rare runtime fallback, not the common case).

Non-TDD plumbing batch (Plan sec 0 ironclad rule 2 explicitly allows this
when test infra would dwarf the change). PA-3 write-path tests will
exercise the helper end-to-end.

Verification: mvn -pl gluten-arrow compile BUILD SUCCESS.

PA-2 batch is now CLOSED. Coverage:
  - PA-2.1 BIGINT scan ........................ TDD GREEN
  - PA-2.2 NaN REAL guard ..................... TDD GREEN
  - PA-2.3 nulls absent ....................... folded into PA-5 sentinel
  - PA-2.4 HUGEINT scan ....................... TDD GREEN
  - PA-2.5a framing layout .................... TDD GREEN
  - PA-2.5b BIGINT statsBlob marshal .......... TDD GREEN
  - PA-2.5c JNI bridge ........................ plumbing (verified by symbol export)
  - PA-2.6 capability helper .................. plumbing (verified by JVM compile)

Backlog (one-line in todos backlog.md):
  - PA-2.5b-real: REAL/Float lower/upperBound 4B LE marshal
  - PA-2.5b-hugeint: HUGEINT/long-Decimal 16B BE marshal

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md

* [GLUTEN-XXXXX][VL] PA-3.0 doc-only: fix scaladoc 'graceful pass-through' claim

The PA-1 scaladoc claimed stats=null falls back to pass-through, but
PA-3.0 investigation (todos rev 2) proved vanilla SimpleMetricsCachedBatchSerializer.buildFilter
NPEs on partitionFilter.eval(null) for non-trivial predicates (both codegen and
interpreted paths, no fallback). PA-3.1 will land the actual lazy-split iterator
wrapper override; this commit only fixes the misleading scaladoc.

refs: todos/features/gluten-inmemory-cache-stats/investigations/0003-simplemetrics-buildfilter-survey.md (rev 2)

* [GLUTEN-XXXXX][VL] PA-3.1 RED: buildFilter stats=null NPE under EqualTo predicate

Flip base class to SimpleMetricsCachedBatchSerializer and remove the pre-PA-3
TODO buildFilter override. Add ColumnarCachedBatchBuildFilterSuite asserting
that v1 binary (stats=null) batches are not dropped when an EqualTo predicate
is applied.

Expected RED failure (observed):
  java.lang.NullPointerException ... isNullAt(int) ... <parameter1> is null
    at GeneratedClass$SpecificPredicate.eval(Unknown Source)
    at SimpleMetricsCachedBatchSerializer.$anonfun$buildFilter$8(CachedBatchSerializer.scala:365)

Matches the prediction in todos investigations/0003 rev 2 evidence 3 exactly:
vanilla SimpleMetricsCachedBatchSerializer.buildFilter NPEs on
partitionFilter.eval(cachedBatch.stats=null) for non-trivial predicates,
both codegen and interpreted paths (no fallback).

PA-3.1 GREEN will add a lazy-split iterator wrapper override that directs
stats=null batches through (skipping vanilla partition pruning) and feeds
stats!=null batches to super.buildFilter. The naive occupier-row placeholder
fails differently (silent drop, see evidence 3.A); only the lazy-split
wrapper is correct.

refs: todos/features/gluten-inmemory-cache-stats/investigations/0003-simplemetrics-buildfilter-survey.md (rev 2)
refs: todos/features/gluten-inmemory-cache-stats/docs/0001-layerA-minmax-design.md (rev 4 D-A3)
refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md PA-3.1

* [GLUTEN-XXXXX][VL] PA-3.1 GREEN: lazy-split iterator wrapper for buildFilter stats=null guard

Override buildFilter with a lazy-split iterator wrapper:
  - stats=null batches (v1 binary read path, PA-4 SQLConf-off write path) are
    directed through unchanged (skipping vanilla partition pruning).
  - stats!=null batches are drained in contiguous runs and routed through
    super.buildFilter for actual partition pruning.

Implementation:
  - cachedBatchIterator.buffered enables stats peek without consuming.
  - Inner runIt sub-iterator self-terminates at the first stats=null boundary,
    so a single parent invocation pulls only the contiguous stats!=null run.
  - staged Iterator buffers parent's output for hasNext accuracy.
  - Lazy throughout: no Iterator materialization, no per-batch allocation
    beyond the sub-iterator wrapper.

Verified:
  - ColumnarCachedBatchBuildFilterSuite 1/1 PASS (PA-3.1 GREEN)
  - ColumnarCachedBatchKryoSuite 3/3 PASS (PA-1 regression sentinel)
  - Total 4/4 PASS, no NPE.

refs: todos/features/gluten-inmemory-cache-stats/investigations/0003-simplemetrics-buildfilter-survey.md (rev 2)
refs: todos/features/gluten-inmemory-cache-stats/docs/0001-layerA-minmax-design.md (rev 4 D-A3)
refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md PA-3.1

* [GLUTEN-XXXXX][VL] PA-3.2 RED: statsBlob binary framing (BIGINT 1-col)

Add ColumnarCachedBatchStatsBlobSuite with 4 cases against the cpp-aligned LE
wire format (cpp/velox/operators/serializer/VeloxColumnarBatchSerializer.cc
PA-2.5b helper). Also bump PA-1.1 test to use the vanilla 5-slot/col
PartitionStatistics schema (lower, upper, nullCount, count, sizeInBytes)
instead of the 2-field placeholder, because PA-3.2 GREEN production change
requires numFields % 5 == 0.

Expected RED failures (observed):
  - PA-3.2.A: statsBlob byte-for-byte vs Java Ser blob -> mismatch, AssertionError
  - PA-3.2.C: corrupt 4-byte blob -> StreamCorruptedException, but we
    expected IllegalArgumentException with 'numCols' guard

PASS in RED (honest sentinel only, will stay GREEN after PA-3.2 GREEN):
  - PA-3.2.B: round-trip InternalRow 5-field (Java Ser handles it accidentally)
  - PA-3.2.D: unsupported col round-trip (Java Ser handles it accidentally)
  - PA-1.1/1.2/1.3 + PA-3.1: full regression sentinel

The honest RED cases (A + C) drive the GREEN: replace Java Serialization
with LE statsBlob marshal aligned to cpp PA-2.5b wire (numCols u32 LE,
per col supported u8 / nullCount u32 LE / count u32 LE / sizeInBytes u64 LE
/ optional lower+upper as u32 len + i64 LE for BIGINT).

refs: todos/features/gluten-inmemory-cache-stats/docs/0002-cpp-stats-contract.md (rev 4 sec 3)
refs: todos/features/gluten-inmemory-cache-stats/docs/0003-jni-binary-framing-reference.md (rev 4 sec 3-4)
refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md PA-3.2

* [GLUTEN-XXXXX][VL] PA-3.2 GREEN: replace Java Ser placeholder with cpp-aligned statsBlob marshal

Replace serializeStats / deserializeStats Java Serialization placeholders
with a binary marshal aligned to cpp PA-2.5b wire format
(cpp/velox/operators/serializer/VeloxColumnarBatchSerializer.cc):

  Layout (LE throughout, no schemaVer prefix to match cpp):
    [ numCols: uint32 LE ]
    per col:
      [ supported: uint8 ]
      [ nullCount: uint32 LE ]
      [ count: uint32 LE ]
      [ sizeInBytes: uint64 LE ]
      if supported:
        [ lowerBoundLen: uint32 LE = 8 ]
        [ lowerBound: int64 LE ]
        [ upperBoundLen: uint32 LE = 8 ]
        [ upperBound: int64 LE ]

PA-3.2 scope: BIGINT 1-col only. Multi-col + other types land in PA-3.3 /
PA-3.5 / follow-up. The InternalRow layout is vanilla 5-slots/col in order
(lowerBound, upperBound, nullCount, count, sizeInBytes) per
investigations/0003 rev 2 evidence 2.

Eager guards:
  - serializeStats: require(numFields % 5 == 0) catches stale 2-field callers.
  - deserializeStats: require(numCols in [0, 4096]) catches truncated /
    corrupt blobs before they NPE downstream.

Verified:
  - ColumnarCachedBatchStatsBlobSuite 4/4 PASS (PA-3.2.A/B/C/D)
  - ColumnarCachedBatchKryoSuite 3/3 PASS (PA-1 regression)
  - ColumnarCachedBatchBuildFilterSuite 1/1 PASS (PA-3.1 regression)
  - 8/8 total ALL PASS, no NPE, no silent corruption.

NOTE: contract docs/0002 sec 3 still mentions a schemaVer:uint8=0x01 prefix
that the cpp end does not emit. Follow-up doc fix in todos PA-3.2 review
batch (separate commit).

refs: todos/features/gluten-inmemory-cache-stats/docs/0002-cpp-stats-contract.md (rev 4 sec 3)
refs: todos/features/gluten-inmemory-cache-stats/docs/0003-jni-binary-framing-reference.md (rev 4 sec 3-4)
refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md PA-3.2

* [GLUTEN-XXXXX][VL] PA-3.3 RED: framed bytes parser test (BIGINT 1-col)

Add ColumnarCachedBatchFramedBytesSuite with 3 cases asserting the JVM-side
parser for the JNI serializeWithStats framed return:

  [ V2_MAGIC: 4 bytes 0xFE 0xCA 0x53 0x02 ]
  [ statsLen: uint32 LE ]
  [ statsBlob: statsLen bytes ]
  [ bytesLen: uint32 LE ]
  [ bytesBlob: bytesLen bytes ]

Tests:
  - PA-3.3.A round-trip stats + bytesBlob through parseFramedBytes
  - PA-3.3.B corrupt magic fails with 'magic' in error
  - PA-3.3.C truncated framing fails eagerly

Expected RED failure (observed): testCompile error
  'value parseFramedBytes is not a member of CachedColumnarBatchKryoSerializer'

Honest RED per AGENTS.md TDD law 3 (failure because feature missing,
not typo). PA-3.3 GREEN will:
  (a) add parseFramedBytes(blob): (InternalRow, Array[Byte]) helper
  (b) wire write path to call jniWrapper.serializeWithStats(handle) and
      parseFramedBytes the return; the wire step itself is plumbing per
      Plan sec 0 law 2 because driving it under unit test needs a native
      lib + real ColumnarBatch fixture, which PA-3.5 e2e will cover.

refs: todos/features/gluten-inmemory-cache-stats/docs/0003-jni-binary-framing-reference.md (rev 4 sec 2)
refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md PA-3.3

* [GLUTEN-XXXXX][VL] PA-3.3 GREEN: parseFramedBytes + write path JNI wiring

(a) Add CachedColumnarBatchKryoSerializer.parseFramedBytes(framed):
    (InternalRow, Array[Byte]) helper. Parses the JNI serializeWithStats
    framed return (cpp PA-2.5c):
      [V2_MAGIC(4) | statsLen u32 LE | statsBlob | bytesLen u32 LE | bytesBlob]
    Eager require() guards catch corrupt magic + truncated framing before
    propagating into a malformed CachedColumnarBatch.

(b) Wire convertColumnarBatchToCachedBatch write path (plumbing, see law 2):
    capability-check via ColumnarBatchSerializerJniWrapper.supportsStatsExt()
    (gluten-arrow PA-2.6 helper, reflection-cached). When supported, call
    jniWrapper.serializeWithStats(handle), parseFramedBytes the return,
    and emit CachedColumnarBatch(stats != null). When unsupported (older
    cpp libgluten.so), fall back to the original serialize() path with
    stats=null; PA-3.1 lazy-split iterator wrapper handles the read side.

The wire step itself is plumbing per Plan sec 0 law 2 because driving it
under unit test needs a native lib + real ColumnarBatch fixture. PA-3.5
e2e will cover it (cache a BIGINT col DataFrame, filter pushdown, assert
partition skip count > 0).

Verified:
  - ColumnarCachedBatchFramedBytesSuite 3/3 PASS (PA-3.3.A/B/C)
  - ColumnarCachedBatchStatsBlobSuite 4/4 PASS (PA-3.2 regression)
  - ColumnarCachedBatchKryoSuite 3/3 PASS (PA-1 regression)
  - ColumnarCachedBatchBuildFilterSuite 1/1 PASS (PA-3.1 regression)
  - 11/11 ALL PASS, no NPE, no silent corruption.

refs: todos/features/gluten-inmemory-cache-stats/docs/0003-jni-binary-framing-reference.md (rev 4 sec 2)
refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md PA-3.3

* [GLUTEN-XXXXX][VL] PA-3.4: lock down lazy-split wrapper prune semantics + fix exhaustion bug

Add ColumnarCachedBatchBuildFilterPruneSuite (3 cases) locking the stats!=null
prune branch of the PA-3.1 lazy-split iterator wrapper -- PA-3.1 only covered
the stats=null direct-through path.

Cases:
  - PA-3.4.A literal in [lower, upper] -> batch kept
  - PA-3.4.B literal outside [lower, upper] -> batch pruned
  - PA-3.4.C mixed null/non-null stream: null through, non-null pruned correctly

PA-3.4.B uncovered a real bug in the PA-3.1 wrapper (test was supposed to be
a regression sentinel but caught a live defect):

  Symptom: java.util.NoSuchElementException: next on empty iterator
    at ColumnarCachedBatchSerializer$$anon$3.next(...:608)

  Root cause: hasNext returned true based on (staged.hasNext || peekable.hasNext)
  but next() recursed and tried to peekable.head after parent fully consumed
  the runIt and pruned all batches -> peekable empty + staged empty -> crash.

  Fix: refactor to advance() helper that loops drain-and-stage until staged has
  a ready element or peekable is exhausted. hasNext / next both call advance
  first, so hasNext is honest and next never sees an empty staged. Also handles
  the case where parent prunes the entire run and the next batch is stats=null
  pass-through (loop continues into the null branch).

  This is exactly why PA-3.4 mattered: PA-3.1 had no test exercising the
  prune-everything path.

Verified: 14/14 PASS (PA-3.4 3 + PA-3.3 3 + PA-3.2 4 + PA-1 3 + PA-3.1 1).

refs: todos/features/gluten-inmemory-cache-stats/docs/0001-layerA-minmax-design.md (rev 4 D-A3)
refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md PA-3.4

* PA-3.5: add ColumnarCachedBatchE2ESuite (e2e smoke)

3 cases under VeloxWholeStageTransformerSuite (true native fixture):
  A. cache + equality filter: no crash + correct result
  B. plan contains InMemoryTableScanExec + ColumnarCachedBatchSerializer kicked in
  C. numOutputRows < N (significantly less than full-scan), <= 2 partitions worth

Precise prune semantics are anchored by ColumnarCachedBatchBuildFilterPruneSuite
(PA-3.4); this suite is e2e smoke only — non-TDD pure regression sentinel.

Local fixture init blocked by NoClassDefFoundError: scala/Serializable at
GlutenBuildInfo on mvn scalatest:test path (classloader); deferred to fork CI.
Reference implementation VeloxColumnarCacheSuite uses the same base class and
is green on community CI, confirming the fixture is sound.

refs: todos/features/gluten-inmemory-cache-stats/docs/0005-pa35-e2e-test-shape-note.md

* PA-4: SQLConf gate spark.gluten.sql.columnar.tableCache.partitionStats.enabled (default false)

Adds default-off ship gate for the Layer A min/max partition stats path:

- gluten-substrait/.../GlutenConfig.scala: new COLUMNAR_TABLE_CACHE_PARTITION_STATS_ENABLED
  entry, key spark.gluten.sql.columnar.tableCache.partitionStats.enabled, default false.
- backends-velox/.../ColumnarCachedBatchSerializer.scala: write path double-gates on
  conf && supportsStatsExt; when either is false falls back to legacy serialize() with
  stats=null. The PA-3.1 lazy-split iterator wrapper transparently passes such batches
  through, so disabling the conf is a pure no-op for read correctness.
- ColumnarCachePartitionStatsConfSuite: 2 cases (key + default-off, doc mentions
  rationale).
- ColumnarCachedBatchE2ESuite: set partitionStats=true so the e2e smoke exercises
  the stats path under fork CI.

Local regression: 16/16 ALL PASS (PA-4 2 + PA-3.4 3 + PA-3.3 3 + PA-3.2 4 + PA-1 3 +
PA-3.1 1). E2E suite still deferred to fork CI for native runtime.

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md PA-4

* PA-3.5: fix numOutputRows lower bound (0 is legal full-prune)

When all batches under InMemoryTableScanExec are pruned by the partition
stats, the numOutputRows metric on that node may legitimately read 0 -- the
surviving row(s) come from a different physical path (Gluten native scan
metrics, or higher-layer pivot resolution). The semantic correctness is
anchored by PA-3.5.A (count == pivot rowcount); this case only needs to
refute a full-scan, not constrain a lower bound.

Local verification: 19/19 ALL PASS (PA-3.5 e2e 3 / PA-4 2 / PA-3.4 3 /
PA-3.3 3 / PA-3.2 4 / PA-1 3 / PA-3.1 1) under
  build/mvn test -pl backends-velox -Pspark-4.1 -Pjava-17 -Pscala-2.13
after a full ./dev/buildbundle-veloxbe.sh + ./build/mvn install pass.

refs: todos/features/gluten-inmemory-cache-stats/docs/0005-pa35-e2e-test-shape-note.md

* test(cache-stats): PA-5 4 ignore as PA-6..PA-10 acceptance criteria

- PA-5.B Float NaN: ignore + investigation 0006 deferred to PA-10
- PA-5.C/D/E Decimal/String marshal: ignore + acceptance criteria for follow-up
  PR (will be flipped to test in PA-7/PA-8/PA-9 when full-type marshal lands)
- 24 case suite: 20 PASS / 4 ignored / 0 FAIL (build/mvn test 1m06)
- spotless auto-applied import order normalization

refs: todos/features/gluten-inmemory-cache-stats/docs/0008-layerA-fulltype-extension.md
refs: todos/features/gluten-inmemory-cache-stats/investigations/0006-pa5b-float-nan-prune.md

* feat(cache-stats): PA-6.0 plumbing schema field + Kryo wire + dispatch params

D-A5 D-A5 plumbing for full-type marshal (no behavior change yet, BIGINT-only):
- CachedColumnarBatch: + schema: StructType (default null, source-compatible)
- Kryo writer/reader: v3 layout adds nullable schema JSON segment after stats
- serializeStats / deserializeStats / parseFramedBytes: add schema parameter
  (unused inside; PA-6.A onwards reads it for dataType dispatch)
- Write path: convert Seq[Attribute] to StructType once and carry per-batch
- Tuple destructuring with type ascription causes runtime MatchError under
  Scala 2.13 Tuple2 erasure -- use t._1/t._2 instead

Sentinel: 20 PASS / 4 ignored / 0 fail (no regression)

refs: todos/features/gluten-inmemory-cache-stats/docs/0008-layerA-fulltype-extension.md
refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md (PA-6.0)

* feat(cache-stats): PA-6.A INT round-trip 4B LE dispatch by schema

serializeStats / deserializeStats now branch on schema(col).dataType:
  IntegerType -> 4 LE bytes (writeU32LE bit-pattern == signed Int LE)
  LongType    -> 8 LE bytes (existing PA-3.2 path)
  other       -> UnsupportedOperationException (PA-6.B/C/F/G or PA-7..PA-10)
schema=null keeps legacy BIGINT-only behavior (back-compat for callsites
that have not been updated).

RED log: ClassCastException Integer cannot be cast to Long at
  CachedColumnarBatchKryoSerializer$.serializeStats(...:304)
GREEN: 21 PASS / 4 ignored / 0 fail (full 8-suite sentinel + new
  ColumnarCachedBatchIntFamilyMarshalSuite)

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md (PA-6.A)

* feat(cache-stats): PA-6.B SMALLINT round-trip 2B LE dispatch

- serializeStats: + ShortType branch using new writeU16LE helper
- deserializeStats: + ShortType branch reading buf.getShort
- writeU16LE helper added next to writeU32LE/writeU64LE

RED: UnsupportedOperationException 'dispatch for ShortType not implemented'
GREEN: 22 PASS / 4 ignored / 0 fail (full sentinel + new PA-6.B case)

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md (PA-6.B)

* feat(cache-stats): PA-6.C TINYINT round-trip 1B dispatch

- serializeStats / deserializeStats: + ByteType branch (1 byte signed)

RED (verified by stash + isolated run):
  PA-6.C TINYINT round-trip 1B preserves value (incl negative) *** FAILED ***
  java.lang.UnsupportedOperationException: PA-6.A serializeStats: dispatch for
    ByteType not implemented yet (landing in PA-6.B/C/F/G or PA-7..PA-10)
  at CachedColumnarBatchKryoSerializer$.serializeStats(...:330)

GREEN: 23 PASS / 4 ignored / 0 fail (full 9-suite sentinel)

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md (PA-6.C)

* feat(cache-stats): PA-6.F YearMonth/DayTime interval shares INT/LONG dispatch

- IntegerType  | _: YearMonthIntervalType => 4 LE bytes (months as Int)
- LongType     | _: DayTimeIntervalType   => 8 LE bytes (microseconds as Long)
Aligned with vanilla ColumnBuilder.scala line 187-189 union pattern.

RED log:
  PA-6.F.1 YearMonthInterval round-trip 4B LE *** FAILED ***
    UnsupportedOperationException: dispatch for YearMonthIntervalType(0,1)
  PA-6.F.2 DayTimeInterval   round-trip 8B LE *** FAILED ***
    UnsupportedOperationException: dispatch for DayTimeIntervalType(0,3)
GREEN: 25 PASS / 4 ignored / 0 fail (full sentinel)

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md (PA-6.F)

* feat(cache-stats): PA-6.G Date / Timestamp / TimestampNTZ dispatch (share INT/LONG)

Aligned with vanilla Spark ColumnBuilder.scala line 187-189:
  IntegerType | DateType | _: YearMonthIntervalType => 4 LE bytes
  LongType    | TimestampType | TimestampNTZType
    | _: DayTimeIntervalType                         => 8 LE bytes
TimeType deferred (spark-4.1 may not expose on this code path).

RED log:
  PA-6.G.1 Date         *** FAILED *** UnsupportedOperationException: DateType
  PA-6.G.2 Timestamp    *** FAILED *** UnsupportedOperationException: TimestampType
  PA-6.G.3 TimestampNTZ *** FAILED *** UnsupportedOperationException: TimestampNTZType
GREEN: 28 PASS / 4 ignored / 0 fail (full sentinel)

refs: todos/features/gluten-inmemory-cache-stats/docs/0004-layerA-implementation-plan.md (PA-6.G)

* [gluten-cache-stats][PA-6.2] cpp computeStats + framedSerializeWithStats typeKind dispatch INT/SMALLINT/TINYINT

- computeStats: add INTEGER/SMALLINT/TINYINT cases (int32/16/8 min/max via scanMinMax)
- framedSerializeWithStats: replace BIGINT-only gate with typeKind dispatch
  - BIGINT: 8B LE (existing)
  - INTEGER: 4B LE
  - SMALLINT: 2B LE (new pushU16LE helper)
  - TINYINT: 1B
- Wire widths align with JVM dispatch (PA-6.A/B/C)
- Tests: PA_6_2_A_1/2/3 RED -> GREEN

refs: todos/features/gluten-inmemory-cache-stats/docs/0008-d-a5-full-type-alignment.md

* [gluten-cache-stats][PA-6.2.D] Date e2e prune correctness via INTEGER stats

Verification test for PA-6.2.B cpp INTEGER computeStats + 4B LE marshal.
DateType column (days-since-epoch Int) prunes end-to-end same as BIGINT:
- no crash
- result count correct (1 row)
- numOutputRows <= upperBound (full-scan refuted)

E2E suite: 5/5 PASS, 1 ignored (PA-5.B NaN deferred).

refs: todos/features/gluten-inmemory-cache-stats/docs/0008-d-a5-full-type-alignment.md

* [gluten-cache-stats][PA-6.5] hotfix Layer-2 mid-review BLOCKERS

Fixes 4 BLOCKERS + Accept bucket from 3-persona Layer 2 mid-review of
PA-1..PA-6.2.D (reviews/batch-pa6.2-mid-review.md).

BLOCKERS:
- B1: schema(base) -> schema(col) in serializeStats/deserializeStats.
  base = col*5 was indexing into expanded PartitionStatistics row; passed
  schema has only numCols entries. Multi-column cache now no-crash.
- B2: JVM-side dispatch allowlist (isDispatchable). cpp may emit
  supported=1 for short-Decimal (Velox BIGINT physical) and other types
  not yet in JVM dispatch (Decimal/String/Bool/Float/Double); JVM gates
  in serializeStats and skips payload in deserializeStats instead of
  throwing UOE. Decimal/String/Bool land in PA-7..PA-10.
- B3: count = numRows - nullCount per vanilla PartitionStatistics
  (was numRows for all cols).
- B4: drop HUGEINT case in cpp computeStats (emit_gate excluded it,
  leaving dead code; restored in PA-8).

D1 (V3_MAGIC bump):
- V3_MAGIC = 0xFE CA 53 03 distinguishes schema-tail-bearing layout from
  pre-PA-6.0 V2 (no schema tail).
- readV3 (PA-6.0 layout, current writer) + readV2NoSchema (rolling-deploy
  compat: old executor stream still parsable; schema=null falls back to
  legacy BIGINT-only behavior).

Accept bucket:
- H1: parseFramedBytes bytesLen == buf.remaining() strict equality
  (was <=, allowing trailing garbage).

Tests:
- E2E suite +2: PA-6.5 B1 multi-column 3-col Long cache (no IOOB),
  PA-6.5 B2 Decimal column (no UOE).
- All 9 ColumnarCach* suites: 35/35 PASS, 4 ignored (PA-5.B/C/D/E
  deferred to PA-7..10).
- cpp gtest 8/8 PASS (PA_2_4 HUGEINT updated to assert unsupported).

refs: todos/features/gluten-inmemory-cache-stats/reviews/batch-pa6.2-mid-review.md

* [gluten-cache-stats][PA-7] short-Decimal (precision <= 18) marshal via 8B unscaled Long

- isDispatchable: add DecimalType(p<=18) allowlist case
- serializeStats: short-Decimal -> 8B LE unscaled Long via Decimal.toUnscaledLong
- deserializeStats: 8B LE -> Decimal.apply(unscaled, p, s)
  T1 trap (per cross-system-marshal-traps.md): MUST wrap as Decimal,
  not raw Long, to avoid SpecificInternalRow.getDecimal ClassCastException.
- Flip PA-5.C (was ignored ship blocker placeholder) to live PA-7 round-trip
  test; passes schema=DecimalType(10,2) so dispatch reaches Decimal arm.
- cpp side unchanged: Velox short-decimal physical = BIGINT, already emits
  8B LE unscaled (PA-2.5b/PA-6.2.B BIGINT path).
- Long-decimal (precision > 18) still deferred to PA-8.

Tests: 32/32 PASS, 3 ignored (PA-5.B NaN / PA-5.D long-Decimal / PA-5.E String).

refs: todos/features/gluten-inmemory-cache-stats/docs/0008-d-a5-full-type-alignment.md

* [gluten-cache-stats][PA-8] long-Decimal (precision > 18) marshal via 16B LE int128

- cpp computeStats: restore HUGEINT case (reverted PA-6.5 B4 drop) for
  long-decimal scan into int128 lo/hi.
- cpp framedSerializeWithStats: HUGEINT arm emits 16 LE bytes
  (low uint64, high uint64).
- JVM isDispatchable: add DecimalType(p<=38) case after p<=18 short arm.
- JVM serializeStats: writeI128LE = pad signed BE BigInteger.toByteArray()
  to 16, sign-extend, reverse to LE.
- JVM deserializeStats: readI128LE = reverse 16B LE -> signed BE
  BigInteger -> Decimal.apply(BigDecimal(bi, scale), p, s).
- Flip PA-5.D ignore to live PA-8 round-trip test (Decimal(30,5) with
  big positive + negative bounds).
- Restore cpp PA_2_4 test assertion (HUGEINT now supported).

Tests: 35/35 PASS, 2 ignored (PA-5.B NaN / PA-5.E String).
cpp gtest 8/8.

refs: todos/features/gluten-inmemory-cache-stats/docs/0008-d-a5-full-type-alignment.md

* [gluten-cache-stats][PA-8.5] hotfix Layer-2 #2 SB1: count semantics inverted

Layer 2 #2 caught: PA-6.5 B3 'fix' was based on wrong premise about vanilla
PartitionStatistics.count semantics. Vanilla ColumnStats.gatherNullStats
(spark/.../ColumnStats.scala:65) increments count even on null rows, so
vanilla count = numRows (total), NOT numRows - nullCount.

Vanilla buildFilter IsNotNull predicate (CachedBatchSerializer.scala:300):
  statsFor(a).count - statsFor(a).nullCount > 0   //   numRows - nullCount > 0

PA-6.5 B3 emitted count = numRows - nullCount, turning the predicate into
  (numRows - nullCount) - nullCount > 0  =  numRows - 2*nullCount > 0

For partitions with nullCount > numRows/2 this silently false-negatives,
DROPPING entire partitions including non-null rows. Silent wrong result.

Fix: cpp emits count = numRows (matches vanilla).

Tests:
- New PA-8.5 SB1 e2e regression in ColumnarCachedBatchE2ESuite tests
  IsNotNull on a 33%-null cache (won't trigger the >50% bug threshold but
  validates the mechanism).
- All 9 ColumnarCach* suites: 36/36 PASS, 2 ignored.

Layer 2 #2 also flagged 5 nits (cpp signed __int128 shift portability,
V3_MAGIC scaladoc stale, parseFramedBytes magic comment, V3 rolling-deploy
note, asymmetric width assert) and 4 medium issues (case other length
sanity, Decimal precision overflow guard, getDecimal scale rescale,
negative DecimalType precision) — tracked to backlog, not fixed in this
hotfix to keep scope tight.

refs: todos/features/gluten-inmemory-cache-stats/reviews/batch-pa8-mid-review.md (pending)

* [gluten-cache-stats][PA-10] Float (4B) + Double (8B) + Boolean (1B) marshal + flip PA-5.B NaN guard

- cpp computeStats: add DOUBLE + BOOLEAN cases (REAL already present from PA-2).
  scanMinMax<float/double> NaN guard at line 124 returns supported=false on
  first NaN -> emit_gate writes supported=0 -> JVM treats as unsupported,
  no silent prune.
- cpp framedSerializeWithStats emit_gate: add REAL/DOUBLE/BOOLEAN.
- cpp wire arms: REAL = 4B IEEE 754 bits via memcpy; DOUBLE = 8B;
  BOOLEAN = 1B (0/1).
- JVM isDispatchable + serializeStats + deserializeStats:
  FloatType (4B Float.intBitsToFloat round-trip),
  DoubleType (8B Double.longBitsToDouble),
  BooleanType (1B).
- Flip PA-5.B (was deferred ignore) to live PA-10 NaN e2e: cache Float
  column with id=7 -> NaN sprinkled, query col=42.0 must still find 1 row.

Tests: 35 PASS / 0 fail / 1 ignored (PA-5.E String only).
cpp gtest 8/8 unchanged.

refs: todos/features/gluten-inmemory-cache-stats/docs/0008-d-a5-full-type-alignment.md

* PA-9: JVM-side String marshal (256B truncate + carry-overflow demote)

- isDispatchable: add StringType
- serializeStats: pre-compute String payload so carry-overflow can demote
  supported=false BEFORE the supported byte is written
- deserializeStats: StringType arm reads lowerLen in [0,256] + UTF8String
- encodeStringBounds helper: 256B truncate + last-byte +1 carry; returns
  None on all-0xFF prefix overflow
- ShipBlockerMarshalSuite: flip PA-5.E ignore -> 3 tests
  (round-trip with CJK bytes / 256B truncation widening / carry-overflow demote)
- cpp VARCHAR scan path deferred to PA-9b (StringView lifetime trap);
  current cpp emits supported=0 for VARCHAR so JVM never sees String
  lo/hi in production -- this PR ships JVM marshal capability ready
- scalastyle: replaced sec sign with 'sec' (memory: non-ASCII black list
  extends beyond em-dash)

Tests: 9-suite sentinel 38/38 PASS / 0 ignored (was 35+1 pre-PA-9).

refs: todos/features/gluten-inmemory-cache-stats/docs/0008-layerA-fulltype-extension.md

* PA-6.2.E: cpp TIMESTAMP scan + emit (toMicros -> 8B LE int64)

RED step: 2 new gtest cases in VeloxColumnarBatchSerializerTest:
  - PA_6_2_E_testComputeStatsTimestampFlatVector
  - PA_6_2_E_2_testFramedSerializeWithStatsTimestamp
Failure modes: hasLowerBound=false (default branch), supported=0,
lowerLen=76 (gibberish past default), confirming feature-missing not typo.

GREEN step: VeloxColumnarBatchSerializer.cc adds
  - TypeKind::TIMESTAMP case in computeStats (scanMinMax<Timestamp> via
    defaulted operator<=>; variant<Timestamp> first-class scalar)
  - emit_gate inclusion of TIMESTAMP
  - framedSerializeWithStats TIMESTAMP arm: 8B LE via toMicros() so the
    JVM LongType path consumes it unchanged (Spark TimestampType /
    TimestampNTZType physical = Long microseconds; vanilla
    ColumnBuilder.scala:188 already aliases them to LONG path).

JVM side unchanged - LongType arm already accepts Timestamp/TimestampNTZ.

Peer audit basis:
  - Vanilla Spark columnar/ColumnBuilder.scala:188 + ColumnType.scala:872
    LongType | TimestampType | TimestampNTZType | DayTimeIntervalType | TimeType
    share the LONG path (8B us). Zero Timestamp-specific stats code.
  - Velox Timestamp.h:377-378 defaulted operator==/<=> -> scanMinMax<>
    compiles directly.
  - Velox Variant.h:259/497 VELOX_VARIANT_SCALAR_CONSTRUCTOR(TIMESTAMP)
    + variant.value<Timestamp>() supported.
  - Gluten VeloxRowToColumnarConverter.cc:162 already uses
    Timestamp::fromMicros(value) as the canonical Spark<->Velox bridge.

Tests: cpp gtest 11/11 PASS (was 9/9 pre-PA-6.2.E). JVM 9-suite
regression unchanged (38/38 PASS).

refs: todos/features/gluten-inmemory-cache-stats/docs/0008-layerA-fulltype-extension.md
      (closes PA-6.2.E deferral)

* PA-6.2.E e2e Timestamp prune sentinel

End-to-end regression sentinel for the cpp TIMESTAMP scan + emit landed
in 9c9a61e8de. Mirrors PA-6.2.D Date pattern:
  N=1000 rows, timestamp_seconds(1704067200 + id), 5 range partitions,
  pivot at N/2. Asserts (a) result == 1, (b) InMemoryTableScanExec in
  plan, (c) numOutputRows <= 2 * (N/P) = 400.

This is a non-TDD regression sentinel (cpp + JVM units already GREEN);
its job is to catch wire/marshal edge cases that unit tests miss and
to preempt the 'have you e2e tested?' reviewer question.

Tests: ColumnarCachedBatchE2ESuite 10/10 PASS (was 9/9).

refs: todos/features/gluten-inmemory-cache-stats/docs/0008-layerA-fulltype-extension.md

* PA-9 cpp: VARCHAR scan + emit + e2e prune sentinel

Reverses prior PA-9b deferral. Peer-impl audit found StringView 'lifetime
trap' was a misread:
  - StringView::operator<=> (StringView.h:219) defaulted, uses memcmp =
    unsigned byte order, matching Spark ByteArray.compareBinary
    (& 0xFF, ByteArray.java:67-90).
  - variant(std::string{sv}) (Variant.h:232) heap-allocates owned string,
    so post-computeStats lifetime is decoupled from RowVector buffer.
  - scanMinMax<StringView> compiles via existing template, no new cmp.

RED step: 2 gtest cases
  - PA_9_testComputeStatsVarcharFlatVector
    Failure: hasLowerBound=false (default branch); high-bit byte 0xc2
    test data confirms unsigned byte ordering matters.
  - PA_9_2_testFramedSerializeWithStatsVarchar
    Failure: lowerLen=0x02 0x00 0x00 0x00 (gibberish past default),
    lower bytes 'apple' missing.

GREEN step: VeloxColumnarBatchSerializer.cc
  - computeStats TypeKind::VARCHAR case: scanMinMax<StringView>, copy
    to owned std::string via variant(std::string(data, size)).
  - emit_gate: include VARCHAR.
  - framedSerializeWithStats VARCHAR arm: pushU32(len) + raw bytes.
    cpp does NOT truncate; JVM PA-9 truncates to 256B on read with
    carry-overflow demote (design 0008 sec 3.1, single source of truth).

E2E sentinel: ColumnarCachedBatchE2ESuite.PA-9 String column equality
filter — N=1000 keys 'k_0000'..'k_0999', 5 range partitions, pivot
'k_0500'. Asserts result==1, InMemoryTableScanExec in plan,
numOutputRows <= 2*(N/P) = 400.

Tests:
  - cpp gtest 13/13 PASS (was 11)
  - JVM 9-suite 38/38 PASS (unchanged, JVM 0 changes)
  - ColumnarCachedBatchE2ESuite 11/11 PASS (was 10)

cpp pitfall fixed mid-flight (sticky for projects/gluten.md):
  string literal '\xc2\xa9copy' parsed as '\xc2', '\xa9c' (greedy hex,
  out-of-range). Fix: split with concatenation '\xc2\xa9' 'copy'.

Layer A type matrix now FULLY closed: Int family / Date / YearMonth /
DayTime / Timestamp / TimestampNTZ / Decimal short+long / Float /
Double / Bool / String — both JVM and cpp sides, all e2e prune
sentinels GREEN.

refs: todos/features/gluten-inmemory-cache-stats/docs/0008-layerA-fulltype-extension.md
      (closes PA-9b deferral)

* CB-1: TIMESTAMP nanos conservative widen (floor lo, ceil hi)

Layer 2 mid-review #3 BLOCKER (cross-persona consensus: PMC#5 + skeptic SB-1).

Velox/Timestamp.h:183 toMicros() = seconds * 1e6 + nanos / 1000 (integer
floor). Naive use in PA-6.2.E floored BOTH lo and hi:
  - floor(lo) is OK: widens prune interval downward, conservative
  - floor(hi) SHRINKS prune interval -> false-negative drops rows whose
    true timestamp has nanos%1000 != 0 (Iceberg/Parquet/Arrow source
    data is naturally ns precision)

Fix: floor(lo), ceil(hi). ceil = floor + 1 us when nanos%1000 != 0.

RED:
  CB_1_testTimestampNanosCeilUpperFloorLower — Timestamp(s, 999500),
    failure: hiMicros=999 want 1000 (toMicros floors away the residue)
  CB_1_2_testTimestampExactMicrosNoCeil — regression sentinel,
    Timestamp(s, 1000) exact 1us must NOT over-ceil

GREEN: framedSerialize TIMESTAMP arm, +1 us when getNanos() % 1000 != 0.

Tests: cpp gtest 15/15 (was 13), JVM 9-suite 38/38 + E2ESuite 11/11
unchanged (JVM 0 changes, wire shape unchanged).

refs: todos/features/gluten-inmemory-cache-stats/reviews/batch-pa9-synthesis.md (CB-1)

* CB-2: cpp VARCHAR truncate to 256B at source (single source of truth)

Layer 2 mid-review #3 BLOCKER (cross-persona consensus PMC#2 + skeptic
SB-5 + perf#5).

Before: cpp emitted full string bytes; JVM truncated to 256B + carry +
demote on read. Two sources of truth, JNI wire bloat (6.4MB/partition
for 64KB strings), and vanilla parity gap not declared.

After: cpp truncates at SOURCE (kStatsStringTruncateLen=256). JVM keeps
encodeStringBounds as defense-in-depth for legacy V2 fallback; JVM
carry path never triggers in production cpp+JVM path. Wire bytes
capped at 256B per bound regardless of source string size.

RED:
  CB_2_testVarcharCppTruncates256B — 300B 'a'/'m', expect lo=256B 'a',
    hi=255B 'm'+'n' (carry); failure: cpp emitted full 300B + wire
    layout offset shifted.
  CB_2_2_testVarcharCppCarryOverflowDemote — 300B 0xFF, expect
    supported=0; failure: cpp emitted supported=1 + raw bytes.

GREEN: VeloxColumnarBatchSerializer.cc computeStats VARCHAR case adds
  truncate + carry + demote. variant(std::move(...)) avoids redundant
  string copy. Mirrors JVM PA-9 encodeStringBounds semantics.

CB_2.3 regression sentinel: short string ('apple'/'banana') round-trip
  unchanged (no accidental rewrite of <= 256B inputs).

Tests: cpp gtest 18/18 PASS (was 13). JVM 9-suite 38/38 + E2ESuite 11/11
unchanged (JVM 0 changes; defense-in-depth carry path still tested by
ColumnarCacheShipBlockerMarshalSuite PA-9 cases).

refs: todos/features/gluten-inmemory-cache-stats/reviews/batch-pa9-synthesis.md (CB-2)
      todos/features/gluten-inmemory-cache-stats/docs/0008-layerA-fulltype-extension.md sec 3.1.a

* S-1a: import + drop body FQDNs in ColumnarCachedBatchSerializer.scala

100 FQDN occurrences in body collapsed to short names via imports:
  - org.apache.spark.sql.types._ (covers all 17 type FQDNs)
  - GenericInternalRow / UTF8String
  - JBigDecimal / BigInteger / JFloat / JDouble (java.lang/math)
  - ByteBuffer / ByteOrder / Arrays / UTF_8 / ByteArrayOutputStream

Pure mechanical rename, zero behavior change. 9-suite 38/38 GREEN.

refs: todos/features/gluten-inmemory-cache-stats/PROGRESS.md

* S-1b-1: tighten top + bottom scaladoc

- CachedColumnarBatch + KryoSerializer scaladoc: drop historical PA-x narrative,
  investigation pointers, and 'will populate later' notes; keep V3 wire layout
  + magic disjointness invariant.
- Bottom 33-line ASCII data-flow pipeline (// format: off block) -> 2-line summary;
  the diagrammed flow is what the methods literally do.

Pure doc cleanup, zero code change. scalastyle + test-compile GREEN.

refs: todos/features/gluten-inmemory-cache-stats/PROGRESS.md

* S-1b-2: tighten Kryo write/read inline comments + extract readOptionalSchema

- Drop PA-x.y / PA-6.5 / 'pre-PA-1' historical inline comments throughout write/
  readV3/readV2NoSchema/readV1; keep only durable invariants (writeInt overload
  warning, +1 NULL distinguisher, Tuple2 Scala 2.13 erasure trap).
- Extract readOptionalSchema helper: dedupes the hasSchema/schemaLen/JSON parse
  block from both readV3 branches (-14 LOC, identical bytes on the wire).

9-suite + E2E 38/38 GREEN.

refs: todos/features/gluten-inmemory-cache-stats/PROGRESS.md

* S-1b-3: tighten object body comments + collapse multi-line case arms

- isDispatchable: per-case PA-x.y attribution comments -> trailing one-liner contracts.
- serializeStats / deserializeStats: drop 4-6 line PA narratives in each case arm,
  keep only durable traps (Decimal-not-Long CCE, cpp-may-emit-supported=1 skip).
- Collapse multi-line 'case A | B | C =>' arms onto single lines (still <= 100 cols).
- Drop dead lowerSkipLen/upperSkip locals in deserialize fallback arm.
- Top dispatch table: 14-line PA-3.2/PA-6.5 B2 history -> 8-line wire layout.
- parseFramedBytes scaladoc: 8 lines -> 4.

Behavior-equivalent (collapse + dead-var removal); 9-suite + E2E 38/38 GREEN.

refs: todos/features/gluten-inmemory-cache-stats/PROGRESS.md

* S-1b-4: tighten buildFilter wrapper + extract statsOf helper

- Drop 9-line PA-3.1 historical narrative (investigation pointers, 'occupier-row
  evidence 3.A' reasoning); keep one-paragraph contract on why the split exists.
- Extract statsOf(batch): InternalRow helper, dedupes the 5-line case match used
  twice (peek for split decision + runIt termination check).
- Collapse hasNext/next bodies to one-liners.

-22 LOC. 9-suite + E2E 38/38 GREEN.

refs: todos/features/gluten-inmemory-cache-stats/PROGRESS.md

* S-2: tighten cpp VeloxColumnarBatchSerializer comments

- .h ColumnStats + computeStats + framedSerializeWithStats: drop PA-2/PA-2.1/PA-2.5
  narrative + 'follow-up slices' notes; keep current contract.
- .cc scanMinMax / per-typeKind arms: drop PA-x.y attribution + reviewer evidence
  (Velox header line numbers, CB-1/CB-2 BLOCKER pointers); collapse to durable
  contract paragraphs (NaN-poisons-column / Timestamp toMicros floor->ceil
  asymmetry / VARCHAR truncate-and-carry).
- framedSerializeWithStats: drop Step 1..4 stepwise headers + per-arm trivial
  comments (REAL = bit reinterpret, BOOLEAN = 1B 0/1).
- Inline pushI128LE local lo/hi vars; collapse trivial blank lines in framed
  assembly tail.

Pure comment/whitespace cleanup, zero behavior change. cpp 18/18 GREEN.

refs: todos/features/gluten-inmemory-cache-stats/PROGRESS.md

* S-3: tighten cpp test PA-x.y RED narratives -> 1-2 line contracts

All 18 TEST_F prologues now state 'what this guards' instead of the
PA-x.y RED rationale + 'Expected RED failure: ... compile error / branch
fall-through' boilerplate, which was useful pre-GREEN but pure noise once
the entire suite is green. PA-x.y test names retained (referenced by
gtest_filter in cron jobs / skills).

cpp gtest 18/18 GREEN.

refs: todos/features/gluten-inmemory-cache-stats/PROGRESS.md

* S-4-1: tighten test suite scaladocs + per-test PA-x.y RED prologues

9 test suites under backends-velox/.../execution/:
- File-top scaladoc: drop PA-1/3.x/4/5/6 attribution + investigations/docs path
  refs + 'Refs:' lists; keep one-paragraph contract on what the suite verifies.
- Per-test inline RED rationale prologues (3+ line // blocks immediately above
  test(/ignore() ): collapse to the first line (now stripped of 'RED:' /
  'GREEN:' prefix). The 'why this fails before GREEN' boilerplate is dead
  weight once the suite is fully green.

Pure comment cleanup, zero behavior change. 9-suite + E2E 40/40 GREEN.

refs: todos/features/gluten-inmemory-cache-stats/PROGRESS.md

* S-4-2: drop body FQDNs in test suites, import short names

11 FQDN body uses across 4 test suites collapsed to short names via imports:
- E2ESuite: java.sql.Timestamp + java.time.Instant + DataFrame
- FramedBytesSuite + ConfSuite: java.util.Locale
- ShipBlockerMarshalSuite: java.util.Arrays

Pure rename, zero behavior change. 9-suite + E2E 40/40 GREEN.

refs: todos/features/gluten-inmemory-cache-stats/PROGRESS.md

* S-5: tighten cpp/Java edge file PA-x.y comments

Three small files touched only by the feature plumbing commits:
- cpp/core/jni/JniWrapper.cc: serializeWithStats JNI entry comment.
- cpp/core/operators/serializer/ColumnarBatchSerializer.h: framedSerializeWithStats
  default impl comment.
- gluten-arrow/.../ColumnarBatchSerializerJniWrapper.java: serializeWithStats
  native + supportsStatsExt helper comments.

Drops PA-2.5c / PA-2.6 attribution + 'PA-3 will...' forward refs; keeps the
durable contract (purpose, fallback semantics, lazy-probe mechanism).

GlutenConfig.scala COLUMNAR_TABLE_CACHE_PARTITION_STATS_ENABLED entry doc
deliberately left unchanged: the doc text is a user-facing SQLConf description
that flows into Spark documentation generation, not internal narrative.

Pure comment cleanup, zero behavior change. cpp 18/18 + JVM compile GREEN.

refs: todos/features/gluten-inmemory-cache-stats/PROGRESS.md

* Strip internal codenames from code body (OSS hygiene)

Removes private POC slugs from all reviewer-visible code:
- inline comments: PA-x.y / CB-1/2 / Layer 2 mid-review BLOCKER attribution
  prefixes and 'design 0008 sec X.Y' cross-doc pointers
- Scala test names: 'PA-3.4.A EqualTo literal in [lower, upper] keeps the batch'
  -> 'EqualTo literal in [lower, upper] keeps the batch' (etc, 50+ tests)
- cpp TEST_F function names: PA_2_1_testComputeStatsLongFlatVector ->
  computeStatsBigintFlatVector (and 16 more)
- gtest assertion messages: drop PA-x.y / CB-1/2 prefix in EXPECT_EQ << "...";

Pure rename / comment cleanup. cpp 18/18 + 9-suite + E2E 40/40 GREEN.

* Rename PA9_STRING_TRUNCATE_LEN -> STRING_BOUND_TRUNCATE_LEN

Identifier carried the phase code glued without dash and slipped past the
prior PA-N regex sweep. Renamed to a name that describes the semantics
(string-bound truncate length) rather than the phase that introduced it.

5 occurrences in ColumnarCachedBatchSerializer.scala. 9-suite + E2E GREEN.

* Drop redundant pre-test() comments in 8 test suites

The S-1b sweep collapsed each per-test PA-x.y RED rationale block to its
first line. After codename stripping that first line was either a verbatim
restatement of the test name ('// EqualTo literal IN range -> batch is
kept.' above test("EqualTo literal in [lower, upper] keeps the batch"))
or a half-sentence stub ('// Carry-overflow demote: upper = 300 bytes of
0xFF cannot widen' before test("String carry overflow demotes column to
unsupported")) -- pure noise.

-68 lines across 8 suites; 9-suite + E2E 40/40 GREEN.

* Drop ColumnarCachePartitionStatsConfSuite

The two cases (conf key + default-off, doc mentions rationale) over-test a
trivial SQLConf entry: the key string and default value are read straight
from the same constant the production code uses (tautological), and the
'doc mentions benchmark/regression' assertion is brittle wording lock-in
that a reviewer would rightly nit. End-to-end conf=true / conf=false
behavior is anchored by ColumnarCachedBatchE2ESuite.

* Drop Kryo wire versioning (V1/V2/V3 magic prefix)

The CachedColumnarBatch Kryo wire never needs cross-version readability:
in-memory cache lifecycle = single SparkSession; spill files die with the
executor; cache() is not persisted across application restarts; no external
service consumes CachedColumnarBatch bytes. The V1/V2/V3 magic-prefix
sniff was defensive code for a scenario that cannot occur in production.

Removes:
- V2_MAGIC / V3_MAGIC constants from the Kryo path
- 4-byte first-byte sniff in read()
- readV1 / readV2NoSchema branches (dead code)
- KryoSuite 'V1 binary backwards-compat' + 'V1 magic-prefix guards against
  silent corruption' tests (~80 LOC of disjointness reasoning)
- All 'rolling-deploy compat' / 'magic disjointness' narrative

Renames the surviving magic to STATS_FRAMED_MAGIC and re-documents it: it
is the cpp/JVM ABI sanity check on the framed JNI return (serializeWithStats),
not a version tag. cpp emit bugs would otherwise feed garbage into
length-prefix readers downstream.

Net -160 LOC; cpp 18/18 + 8-suite + E2E GREEN.

* Float/Double boundary contract: +/-Inf, +/-0, subnormal not poisoned

scanMinMax NaN guard already returns supported=false on any NaN observation.
Document explicitly that the other IEEE 754 corner cases do NOT poison the
column: +/-Infinity participates in min/max as ordered values; +0 and -0
are equal under <; subnormals (denormals) are ordered like normal floats.

Adds cpp gtest computeStatsFloatBoundaryValuesNotPoisoned exercising REAL
and DOUBLE paths with {-Inf, +0, -0, subnormal, +Inf} mixed -- expects
supported=true with lower=-Inf, upper=+Inf.

cpp gtest 19/19 GREEN.

* Fail-fast on unknown-arm upperSkipLen wire bound

The unknown-arm path in deserializeStats reads upperSkipLen as a raw int
and previously allocated new Array[Byte](upperSkipLen) without bounds
checking. A cpp/JVM wire mismatch could surface as an OOM rather than a
clear diagnostic.

Adds require(upperSkipLen in [0, STRING_BOUND_TRUNCATE_LEN]) so the JVM
fails fast with a wire-mismatch hint instead of attempting a multi-GB
allocation. The upper bound matches the cpp-side string truncation SOT.

8-suite 37/37 GREEN.

* Final pre-PR fixes: lowerLen bound, StringType subtype-match, numCols cap

Three correctness/operational fixes from final code review:

1. Hoist require(lowerLen in [0, 256]) immediately after read at L334 so
   it covers ALL dispatch arms (not just the unknown-arm B4 fix). Symmetric
   defense against torn frames or corrupt blobs reaching new Array[Byte](N).

2. Replace eq StringType / case StringType with isInstanceOf[StringType] /
   case _: StringType at the four dispatch sites (isDispatchable, write
   isStringCol guard, write dispatch arm, read dispatch arm). Spark's
   StringType is now class StringType private[sql] with case object
   StringType extends StringType(0); collated columns are class instances
   that fail reference-equality. Without this, every collated string column
   silently loses partition pruning on the read path.

3. Raise the deserializeStats numCols cap from 4096 to Int.MaxValue / 5.
   The 4096 framing as 'corrupt' rejected legitimate wide tables (5k+
   column ML feature stores work fine on vanilla cache). Int.MaxValue / 5
   is the true overflow guard for the GenericInternalRow(numCols * 5)
   allocation that follows.

8-suite 37/37 GREEN.

* Add ColumnarTableCachePartitionStatsBenchmark + results

Quantifies the read-path pruning benefit and the write-path overhead of
the columnar table-cache partition-stats path:

  100M-row range partitioned by c2 into 32 partitions, in-memory cached;
  groupBy + sum/count/avg follow-up to make pruned-partition savings visible.

Results (single-driver local[1], AMD EPYC 7763):

  table cache build (write path)
    partitionStats off    126425 ms       1.0X
    partitionStats on     131431 ms       1.0X    (no measurable overhead)

  filter+agg high selectivity (c2 < 1000, ~0.001%)
    partitionStats off      4431 ms       1.0X
    partitionStats on       1744 ms       2.5X

  filter+agg low  selectivity (c2 < 50000000, ~50%)
    partitionStats off      5332 ms       1.0X
    partitionStats on       3392 ms       1.6X

  filter+agg point lookup (c2 = 50000000, 1 row)
    partitionStats off      4343 ms       1.0X
    partitionStats on       1686 ms       2.6X

* Apply clang-format-15 to cpp sources

Run dev/format-cpp-code.sh against PR sources to satisfy
the CI format-check job (clang-format 15).

* Regenerate docs/Configuration.md via dev/gen-all-config-docs.sh

AllGlutenConfiguration golden-file check requires this whenever a new
SQLConf is added (CI spark-test-* failures pointed here).

* [VL] Replace reflection-based capability check with callsite UnsatisfiedLinkError fallback

The reflection-based supportsStatsExt() helper checks the Java method
declaration via getDeclaredMethod, which always succeeds in this jar
regardless of whether the loaded libgluten.so contains the corresponding
JNI symbol. A new Gluten jar paired with an older native library would
still throw UnsatisfiedLinkError on the first serializeWithStats call,
giving the helper a false sense of safety.

Remove the helper and detect capability lazily at the call site:

- The cache write path queries
  ColumnarCachedBatchSerializer.statsExtAvailable (default true,
  optimistic fast path) before invoking serializeWithStats.
- A try/catch around the call traps UnsatisfiedLinkError once, calls
  markStatsExtUnavailable to flip a JVM-wide volatile flag, and falls
  back to the legacy serialize() path emitting stats=null. Subsequent
  batches in this JVM go straight through the fallback, since a
  jar/native version mismatch is a deployment misconfiguration that
  should not be retried per batch.

The buildFilter wrapper continues to direct stats=null batches through
unchanged, so partition pruning is silently disabled rather than
crashing the executor.

* [VL] Reject oversized serialized batch payload above u32 framing limit

The framedSerializeWithStats wire layout encodes bytesLen as a
u32 LE field, so a single batch payload larger than 4 GiB would
silently truncate when cast from int64_t to uint32_t and produce
a corrupt frame that the JVM-side parser would mis-read.

Add a GLUTEN_CHECK enforcing 0 <= bytesLen <= UINT32_MAX before
the cast. The condition is pathological in practice (would require
a single batch with very wide schemas or huge string columns), but
fail-fast here is cheaper than debugging a corrupt frame d…