Shuffle cleanup for all SQL operations was introduced here: apache/spark@5337a578069. This follow up adds the cleanup also for commands: apache/spark@1dab449da5c
This is something we should test against UCX, since we should in theory see spark clean up buffers from the shuffle catalog as the queries finish. We should also document the issue and potentially recommend config changes for users of UCX, if we see this as a benefit. Either way, we should test UCX with this, and that's the extent of the task (not a code change).