Set restartPolicy and backoff limit, it seems like not effect.

[shaoqingyang]

What happened?

nprocPerNode: auto
pytorchReplicaSpecs:
Master:
replicas: 1
restartPolicy: OnFailure
Worker:
replicas: 1
restartPolicy: OnFailure
runPolicy:
backoffLimit: 1
cleanPodPolicy: All
schedulingPolicy:
priorityClass: job-third
suspend: false
ttlSecondsAfterFinished: 60

What did you expect to happen?

I hope it can effect, maybe my yaml is not correct.

Environment

Kubernetes version:

$ kubectl version

Training Operator version:

$ kubectl get pods -n kubeflow -l control-plane=kubeflow-training-operator -o jsonpath="{.items[*].spec.containers[*].image}"

Training Operator Python SDK version:

$ pip show kubeflow-training

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[shaoqingyang]

[andreyvelich]

Thanks for creating this @shaoqingyang!
Can you try to run this simple PyTorchJob example to see if you are getting the same warning:

training-operator/examples/pytorch/simple.yaml

Line 10 in 265d4c7

restartPolicy: OnFailure

cc @kubeflow/wg-training-leads @Electronic-Waste

[shaoqingyang]

Thanks for creating this @shaoqingyang! Can you try to run this simple PyTorchJob example to see if you are getting the same warning:

training-operator/examples/pytorch/simple.yaml

Line 10 in 265d4c7

restartPolicy: OnFailure

cc @kubeflow/wg-training-leads @Electronic-Waste

I think it can runinng and restart. My problem is that it can retry infinitely without judging based on the number of backoffLimit attempts.

[andreyvelich]

@shaoqingyang Do you see the similar warning in the controller that restartPolicy is not onFailure when you run this example?

[shaoqingyang]

This example has the same issue as the job I created myself. I set both the Master and Worker to RestartPolicy: OnFailure， Then set
runPolicy:
backoffLimit: 3
cleanPodPolicy: All
schedulingPolicy:
priorityClass: job-third
suspend: false
ttlSecondsAfterFinished: 60
The job can restart an unlimited number of times.

[shaoqingyang]

Maybe this counter has not taken effect.

[shaoqingyang]

Sorry, there are some issues with my retry method. I deleted the pod and restarted it. I looked at the code and found that a container exception is required.