[Distributed Processing] Blocking shuffle

[shehabgamin]

With local disks and remote storage

[Enochack]

Is the first version expected to be sort based or hash based?

[shehabgamin]

@Enochack This is something that @linhr is currently exploring so I'll let him share his thoughts.

We would love your input as well though!

[linhr]

We'll likely start with the hash-based approach which is closer to our current streaming shuffle setup. We can then extend to sort-based approaches which would also fit into our unified shuffle design.

If you have thoughts or have use cases to share, we'd love to learn about it!

[linhr]

This is now supported with both the remote storage and Apache Celeborn backends.