enable-gang-scheduling not work

[chenwenjun-github]

What happened:
I use the kubeflow/tf-operator with --enable-gang-scheduling = true, it run on k8s, I have different namespace.
When I submit two same tfjobs at the same time and the resourcequota of namespace is only enough for one tfjob, I find the two tfjob still start to occupy the resources and cause deadlock.

And I try to submit only one tfjob with more resource but the resourcequota of namespace is not enough

I find although podgroup show that the resouce is not enough for all pods of tfjob, but there still have several pod which has been created and pod's status is pending

What you expected to happen:
when the resouce is not enough for all pods of tfjob, all the pods of tfjob shouldn't be created,
because the pod which status is pending still will be distribute resource.

How to reproduce it (as minimally and precisely as possible):
like the describe above

Anything else we need to know?:
tf-operator version is recently either.

Environment:

• Volcano Version:1.2.0
• Kubernetes version (use kubectl version):1.13
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[Thor-wl]

/assign @william-wang Please help take a look

[shinytang6]

xref: kubeflow/trainer#1264

[william-wang]

OK, Let me have a check and investigation in my environment.

[Thor-wl]

According to the discussion of community weekly meeting, adivice as follows:

• switch the use of tf-operator to volcano job api. There will not meet this problem.
• submit an issue to tf-operator to consider gang-scheduling when creating pods.

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

[stale]

Closing for now as there was no activity for last 60 days after marked as stale, let us know if you need this to be reopened! 🤗