Optimize the time for creating pods and services when creating new Job.

[lishangyuzi]

What you would like to be added?

For the creation events of large-scale jobs, such as PyTorch jobs, use multiple goroutines to initiate createPod and createSvc, which can reduce the time for jobs to create Kubernetes (k8s) resources.

Why is this needed?

If there are a large number of pods under a large-scale job, initializing the pods and services (svc) will take a lot of time. This will seriously affect the startup of a large-scale training job.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[lishangyuzi]

The places where pods and svcs are created are as follows. I've tried to modify it to use multiple goroutines for processing.

training-operator/pkg/controller.v1/common/pod.go

Line 321 in d7f69e8

err = jc.createNewPod(job, rt, index, spec, masterRole, replicas)

training-operator/pkg/controller.v1/common/service.go

Line 185 in d7f69e8

err = jc.CreateNewService(job, rtype, spec, strconv.Itoa(index))