[FEA][AUDIT][SPARK-51756][CORE] Computes RowBasedChecksum in ShuffleWriters

[abellina]

This was added to spark 4.1, and it seems to me like a bug fix really but it's not being backported and is off by default:
apache/spark@9ab693c01ed

The issue looks to be in non-deterministic stages where a task failure could result in a recompute producing the wrong values. The idea is to provide a checksum per partition that can be used to figure out if the partition is correct. In case of recomputation, if the partition's checksum had been known before to the driver (via MapStatus) and it doesn't match the new checksum, the stage will be marked as failed.

What would we need to do here? We would need to provide the checksum value as part of MapStatus in our shuffle manager, if we want to take part of this feature. Note that it is off by default.

Alternatively, we could disable the shuffle manager if this feature is turned on, as we don't support it.

[midiakiasat]

This is effectively correctness protection for non-deterministic stages. If RAPIDS shuffle does not propagate RowBasedChecksum via MapStatus, then enabling this Spark feature silently degrades safety. I’d recommend either (a) explicitly wiring checksum support into the RAPIDS shuffle manager, or (b) hard-disabling RAPIDS shuffle when spark.shuffle.checksum.enabled=true, with a clear warning. Silent incompatibility here is risky.

[abellina]

This is effectively correctness protection for non-deterministic stages. If RAPIDS shuffle does not propagate RowBasedChecksum via MapStatus, then enabling this Spark feature silently degrades safety. I’d recommend either (a) explicitly wiring checksum support into the RAPIDS shuffle manager, or (b) hard-disabling RAPIDS shuffle when spark.shuffle.checksum.enabled=true, with a clear warning. Silent incompatibility here is risky.

Thanks @midiakiasat for the comment. I totally agree we need to either fallback or handle it. I am surprised the feature wasn't ported to other spark versions, I guess it must have been enough of an edge case?

I am inclined to fallback for now (should be a trivial change) with that warning and then work on a follow up. The checksum computation would require kernel changes, since we probably want this as a receipt from the partitioning code.

[midiakiasat]

Agreed on fallback as the short-term fix. One concrete suggestion to make this robust:

• When spark.shuffle.checksum.enabled=true, fail fast or force CPU shuffle before job execution (driver-side validation), not lazily at task time.
• Emit a structured warning that includes Spark version + RAPIDS version so misconfigurations are diagnosable from logs.

That keeps correctness guarantees explicit and avoids users unknowingly running without checksum protection. Kernel-level checksum support can then be tracked as a follow-up item.