Is there a sweet spot for choosing the number of coordinator and shuffle service? #1197
-
I'm a newbie of Uniffle. My question might be trivial to others who are already familiar with it. I have following questions that I want to know: 1. Is it OK to have different coordinator quorum number per clients?Let's say client A has 2 quorum, client B has 3 quorum. 2. Is there some kind of sweet spot to choose the optimal number of quorum and shuffle server?Is there an upper asymptote bound of number of quorum? For example, more than 4 or 5 quorum has no difference with 3? 3. How can I find one by myself if it depends on the load.What kind of metrics or logs do I need to look for to get my optimal settings? Please guide me if there is a good way to find the answer. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
Client can use different quorums.It's hard to find suitable quorum number. Usually we choose 1 or 3 quorum number. For important jobs, we can choose 3 quorum. For normal jobs, we can choose 1 quorum. |
Beta Was this translation helpful? Give feedback.
Client can use different quorums.It's hard to find suitable quorum number. Usually we choose 1 or 3 quorum number. For important jobs, we can choose 3 quorum. For normal jobs, we can choose 1 quorum.