[RayCluster] Improved the efficiency when checking rayclusters' expectations

[harryge00]

Why are these changes needed?

The current implementation of IsSatisfied in scale_expectation.go , will iterate through all pods in itemsCache. This is not necessary. When one pod fails to meet the expectation, we can return directly. fast-fail means return earlier once a pod's scaling result is not as expected.

Current code:

	for i := range items {
		rp := items[i].(*rayPod)
		pod := &corev1.Pod{}
		isPodSatisfied := false
		// check if pod is scaled correctly
                ...
		if isPodSatisfied {
			if err := r.itemsCache.Delete(items[i]); err != nil {
				panic(err)
			}
		} else {
			// Here we can return directly. No need to continue.
			// It is inefficient especially when
			// we have created lots of pods and few of them are synced from kube-apiserver
			isSatisfied = false
		}
	}

For example, we may create 10 pods, and expect to successfully get the 10 pods. If none of the pods is ready, no need to iterate through the 10 pods every time. Just find one pod is missing and then break.

I also refactored the code, reduced nested if-else conditions.

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[harryge00]

@Eikykun Since you implemented the first version of RayClusterScaleExpectation, could you view ?

[Future-Outlier]

cc @fscnick @machichima @seanlaii for review if you are interested

for more detail, you can refer #2566

[Future-Outlier]

Suggested change

	if err := r.Get(ctx, types.NamespacedName{Name: rp.name, Namespace: namespace}, pod); err != nil {
	if err := r.Get(ctx, types.NamespacedName{Name: rp.name, Namespace: rp.namespace}, pod); err != nil {

[Future-Outlier]

since we have namespace in rp *rayPod already

Suggested change

	func (r *rayClusterScaleExpectationImpl) isPodScaled(ctx context.Context, rp *rayPod, namespace string) bool {
	func (r *rayClusterScaleExpectationImpl) isPodScaled(ctx context.Context, rp *rayPod) bool {