What would you like to be added:
Please add the pod preferred nodeAffinity to the node score in the HAMi scheduler.
Now the HAMi scheduler picks one node inside Filter. The pod says "I like node A better".
But HAMi does not look at this. HAMi only uses its own GPU score (binpack/spread).
So the pod can go to the wrong node.
The place is OverrideScore in pkg/scheduler/policy/node_policy.go.
We can add a small score when the node matches the pod preferred affinity.
Pods with no preferred affinity do not change at all.
Open questions (we can talk about it):
- how to mix this score with the GPU score
- how big this score should be
- how to set the weight (kube weight 1-100 or a new flag)
Why is this needed:
I see this in my production cluster (HAMi v2.10.0).
My pod had preferred nodeAffinity, weight 100, for node A.
HAMi put the pod on node B. Log:
2 nodes fit(nodeA:0.00,nodeB:9.03)
Successfully bound pod to node ... evaluatedNodes=14 feasibleNodes=1
Both nodes were OK. But node A got a smaller score. So my preferred affinity did nothing.
Now I can only use hard affinity (requiredDuringScheduling). But hard affinity is bad for me:
if node A has no space, my pod stays Pending forever. I want a soft way.
Anything else we need to know?:
What would you like to be added:
Please add the pod preferred nodeAffinity to the node score in the HAMi scheduler.
Now the HAMi scheduler picks one node inside Filter. The pod says "I like node A better".
But HAMi does not look at this. HAMi only uses its own GPU score (binpack/spread).
So the pod can go to the wrong node.
The place is
OverrideScoreinpkg/scheduler/policy/node_policy.go.We can add a small score when the node matches the pod preferred affinity.
Pods with no preferred affinity do not change at all.
Open questions (we can talk about it):
Why is this needed:
I see this in my production cluster (HAMi v2.10.0).
My pod had preferred nodeAffinity, weight 100, for node A.
HAMi put the pod on node B. Log:
Both nodes were OK. But node A got a smaller score. So my preferred affinity did nothing.
Now I can only use hard affinity (requiredDuringScheduling). But hard affinity is bad for me:
if node A has no space, my pod stays Pending forever. I want a soft way.
Anything else we need to know?:
I only want the score to look at preferred affinity. The Filter flow does not change.