[BUG] NDS2.0 query16 validation failure due to unmatched results(intermittent)

[yinqingh]

Describe the bug
Build:

• nds2-parquet-30k-snappy-gh-weekly/39
• nds2-parquet-10k-snappy-sparkh-weekly/42

Query16 validation failed in the third test iteration (202512270151-GPU) while two previous iterations passed successfully. The query execution completed but validation check returned 'Fail' status. This is an intermittent issue affecting query16 in the NDS2.0 30TB parquet benchmark test on Grace Hopper cluster.

Also observed the same failure in nds2-parquet-10k-snappy-sparkh-weekly/42.

Error logs:

[2025-12-27T02:14:15.244Z] Collected 1 rows in 0.08379197120666504 seconds
[2025-12-27T02:14:15.244Z] Row 0: 
[2025-12-27T02:14:15.244Z] [151636, Decimal('634135438.30'), Decimal('-129843205.33')]
[2025-12-27T02:14:15.244Z] [151636, Decimal('634139560.60'), Decimal('-129841602.82')]

Environment details

• Spark Version: 3.4.3
• Hadoop Version: 3.3.6
• Failed Iteration: 3rd run (202512270151)
• Successful Iterations: 1st run (202512270102), 2nd run (202512270126)

[NVnavkumar]

Build nds2-parquet-30k-snappy-gh-weekly/40 also has a similar failure:

[2026-01-03T01:26:52.964Z] Collected 1 rows in 0.06781625747680664 seconds
[2026-01-03T01:26:52.964Z] Row 0:
[2026-01-03T01:26:52.964Z] [151635, Decimal('634139560.60'), Decimal('-129840626.72')]
[2026-01-03T01:26:52.964Z] [151636, Decimal('634139560.60'), Decimal('-129841602.82')]
[2026-01-03T01:26:52.964Z]
[2026-01-03T01:26:52.964Z] Processed 1 rows
[2026-01-03T01:26:52.964Z] There were 1 errors

NDS 2.0 query16 uses a distinct count:

select
   count(distinct cs_order_number) as `order count`
  ,sum(cs_ext_ship_cost) as `total shipping cost`
  ,sum(cs_net_profit) as `total net profit`
from
   catalog_sales cs1
  ,date_dim
  ,customer_address
  ,call_center
...

This accomplished using multiple (Gpu)HashAggregates. From the history server UI, I found an interesting discrepancy between runs in different builds (these are actually both validation errors, but the distinct count is correct in build 39):

Query 49 in build nds2-parquet-30k-snappy-gh-weekly/39

Query 49 in build nds2-parquet-30k-snappy-gh-weekly/40

In build 40, the row count for first GpuHashAggregate is incorrect (151635) and this corresponds to the incorrect distinct count result from the error message. One of the decimal sum aggregates is incorrect in Build 49 (total net profit). Build 39 still has the correct distinct count, but both decimal sum aggregates are incorrect.