[FEATURE] Add longer retry when write connect to shuffleManager to get shuffle handler

[zuston]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the feature

When the partition reassign is enabled, the writer/reader will get the shuffle handler by the grpc with shuffle manager. that means when the driver is under the high pressure of GC, the write will fail due to the grpc timeout

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[zuston]

Although we can use rss.client.rpc.timeout.ms to extend the timeout, this setting applies to both the shuffle server client and the shuffle manager client simultaneously. Given the performance differences between these components, it would be more appropriate to introduce dedicated configuration options to control them independently.

[zuston]

Another point is that the MutableShuffleHandleInfo has the unreasonable data structure.

Normally the partition number will be large but the assigned shuffle-servers will be less than the partition number, so in the current codebase, the grpc proto structure will stored duplicate structure, this will make driver have a large burden, and we could use the following structure to simplify this logic in the grpc proto.

Map<ShuffleServerInfo, List>