Skip to content

[TEST][AUDIT][SPARK-52777][SQL] Enable shuffle cleanup mode configuration in Spark SQL #14084

@abellina

Description

@abellina

Shuffle cleanup for all SQL operations was introduced here: apache/spark@5337a578069. This follow up adds the cleanup also for commands: apache/spark@1dab449da5c

This is something we should test against UCX, since we should in theory see spark clean up buffers from the shuffle catalog as the queries finish. We should also document the issue and potentially recommend config changes for users of UCX, if we see this as a benefit. Either way, we should test UCX with this, and that's the extent of the task (not a code change).

Metadata

Metadata

Assignees

No one assigned

    Labels

    audit_4.1.0Audit related tasks for 4.1.0documentationImprovements or additions to documentationfeature requestNew feature or requesttestOnly impacts tests

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions