bug(MPI Training) : Scheduling Policy doc bug for MPIJob

[ttakahashi21]

This is a Bug Report

Problem:

error unknown field "spec.runPolicy.schedulingPolicy.minResources.reuests.nvidia.com/gpu"

• Error case
  • use requests field

apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  labels:
    kueue.x-k8s.io/queue-name: user-queue
  name: tensorflow-benchmarks
spec:
  slotsPerWorker: 1
  runPolicy:
    cleanPodPolicy: Running
    schedulingPolicy:
      minAvailable: 4
      queue: user-queue
      minResources:
        reuests:
          nvidia.com/gpu: 3
      scheduleTimeoutSeconds: 0
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
         spec:
           containers:
           - image: mpioperator/tensorflow-benchmarks:latest
             name: tensorflow-benchmarks
             command:
             - mpirun
             - --allow-run-as-root
             - -np
             - "2"
             - -bind-to
             - none
             - -map-by
             - slot
             - -x
             - NCCL_DEBUG=INFO
             - -x
             - LD_LIBRARY_PATH
             - -x
             - PATH
             - -mca
             - pml
             - ob1
             - -mca
             - btl
             - ^openib
             - python
             - scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py
             - --model=resnet101
             - --batch_size=64
             - --variable_update=horovod
    Worker:
      replicas: 2
      template:
        spec:
          containers:
          - image: mpioperator/tensorflow-benchmarks:latest
            name: tensorflow-benchmarks
            resources:
              limits:
                nvidia.com/gpu: 1

kubectl apply -f tensorflow-benchmarks.yaml

• Output

Error from server (BadRequest): error when creating "tensorflow-benchmarks.yaml": MPIJob in version "v2beta1" cannot be handled as a MPIJob: strict decoding error: unknown field "spec.runPolicy.schedulingPolicy.minResources.reuests.nvidia.com/gpu"

Proposed Solution:

• Correct case
  • do not use requests field

apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  labels:
    kueue.x-k8s.io/queue-name: user-queue
  name: tensorflow-benchmarks
spec:
  slotsPerWorker: 1
  runPolicy:
    cleanPodPolicy: Running
    schedulingPolicy:
      minAvailable: 4
      queue: user-queue
      minResources:
          nvidia.com/gpu: 3
      scheduleTimeoutSeconds: 0
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
         spec:
           containers:
           - image: mpioperator/tensorflow-benchmarks:latest
             name: tensorflow-benchmarks
             command:
             - mpirun
             - --allow-run-as-root
             - -np
             - "2"
             - -bind-to
             - none
             - -map-by
             - slot
             - -x
             - NCCL_DEBUG=INFO
             - -x
             - LD_LIBRARY_PATH
             - -x
             - PATH
             - -mca
             - pml
             - ob1
             - -mca
             - btl
             - ^openib
             - python
             - scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py
             - --model=resnet101
             - --batch_size=64
             - --variable_update=horovod
    Worker:
      replicas: 2
      template:
        spec:
          containers:
          - image: mpioperator/tensorflow-benchmarks:latest
            name: tensorflow-benchmarks
            resources:
              limits:
                nvidia.com/gpu: 1

kubectl apply -f tensorflow-benchmarks.yaml

-Output

mpijob.kubeflow.org/tensorflow-benchmarks created

Page to Update (provide the full path):

https://www.kubeflow.org/docs/components/trainer/legacy-v1/user-guides/mpi/#scheduling-policy

Component/Kubeflow Version:

• kubeflow : v2beta1
• MPIjob : https://raw.githubusercontent.com/kubeflow/mpi-operator/master/deploy/v2beta1/mpi-operator.yaml

Labels

/area website
/area trainer

---

Impacted by this bug? Give it a 👍.

[ttakahashi21]

/assign ttakahashi21

[ttakahashi21]

/area trainer