[MultiKueue] Support Deployment Integration

[Bobbins228]

What would you like to be added:

The ability to create Kubernetes deployments on remote Clusters through MultiKueue
Why is this needed:

Support for various integrations already exist i.e. Jobset, KubeFlow Jobs, MPI, Batch.
Our use case is for long running model serving deployments that can be created remotely from a Manager Cluster.

Completion requirements:

Deployments can be created/managed locally on the Manager Cluster and through MultiKueue created/managed on the Worker Cluster(s) without the risk of running on the Manager.

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[mimowo]

cc @mwielgus @mwysokin

[tenzen-y]

In the ideal solution, I think that we need to implement the managedBy field in all workload objects like Deployment and StatefulSet.

@mimowo Do you have any concerns about implementing the managedBy feature same as the batch/v1 Job?

[mimowo]

We don't know yet the best path forward, from the initial discussion with @mwielgus we would like to support MultiKueue for Pods. Then, Deployement and StatefulSet integrations would work for free. The users using pod integration could also benefit. The problem is how to achieve "managedBy" for pods. The initial ideas we discussed:

1. gate the pod on the management cluster and block status updates
2. gate the pod on the management cluster and support status updates
3. schedule the pod on the management cluster, but on a virtual MultiKueue node

1, might will work ok, but might not be transparent to the end-users. 2 is relatively simple technically, but the issue is that we update the status, while the pod is gated, which might be violating the Pod API. 3. is harder, but would allow to update the Pod status without tricks.