autopilot-clusters.md

Set Up Clusters

Manager Cluster

This cluster will act as the central control plane for ingesting jobs and dispatching them to the worker clusters through MultiKueue. It's recommended that this cluster doesn't have any node pools with GPUs, as its primary role is orchestration, not job execution.

gcloud container --project "gpu-launchpad-playground" clusters create-auto "mwy-multikueue-manager" --region "us-central1"

Worker Clusters

These are the clusters where your jobs will actually run.

gcloud container --project "gpu-launchpad-playground" clusters create-auto "mwy-multikueue-worker1" --region "us-central1"
gcloud container --project "gpu-launchpad-playground" clusters create-auto "mwy-multikueue-worker2" --region "us-east4"

Get Cluster Credentials

Obtain the kubeconfig credentials for your clusters to interact with them.

gcloud container clusters get-credentials mwy-multikueue-manager --region us-central1 --project gpu-launchpad-playground
KUBECONFIG=worker-1.kubeconfig gcloud container clusters get-credentials mwy-multikueue-worker1 --region us-central1 --project gpu-launchpad-playground
KUBECONFIG=worker-2.kubeconfig gcloud container clusters get-credentials mwy-multikueue-worker2 --region us-east4	--project gpu-launchpad-playground

Install Kueue

• Install Kueue on each cluster.
• To simplify the setup, disable integrations for AppWrapper, KubeRay, and Kubeflow

export KUBECONFIG=worker-1.kubeconfig
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.12.1/manifests.yaml
kubectl wait --for=condition=available --timeout=600s deployment/kueue-controller-manager -n kueue-system
kubectl get -n kueue-system configmap kueue-manager-config -o yaml | egrep -v '(ray.io|kubeflow.org|codeflare.dev)' | kubectl apply --server-side -f -
kubectl delete po --all -n kueue-system

export KUBECONFIG=worker-2.kubeconfig
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.12.1/manifests.yaml
kubectl wait --for=condition=available --timeout=600s deployment/kueue-controller-manager -n kueue-system
kubectl get -n kueue-system configmap kueue-manager-config -o yaml | egrep -v '(ray.io|kubeflow.org|codeflare.dev)' | kubectl apply --server-side -f -
kubectl delete po --all -n kueue-system

unset KUBECONFIG
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.12.1/manifests.yaml
kubectl wait --for=condition=available --timeout=600s deployment/kueue-controller-manager -n kueue-system
kubectl get -n kueue-system configmap kueue-manager-config -o yaml | egrep -v '(ray.io|kubeflow.org|codeflare.dev)' | kubectl apply --force-conflicts --server-side -f -
kubectl delete po --all -n kueue-system

Manager Cluster

On the Manager Cluster additionally enable the MultiKueue featuregate

kubectl patch deployment kueue-controller-manager -n kueue-system --patch \
    '{"spec":{"template":{"spec":{"containers":[{"name":"manager","args":["--config=/controller_manager_config.yaml","--zap-log-level=8","--feature-gates=MultiKueueBatchJobWithManagedBy=true"]}]}}}}'

---

Create MultiKueue Service Account + Role Bindings on Worker Clusters

This script creates the necessary service account and role bindings on your worker clusters, allowing the MultiKueue manager to interact with them. It then outputs a kubeconfig that uses that serviceaccount to communicate with the worker clusters.

wget -N https://kueue.sigs.k8s.io/examples/multikueue/create-multikueue-kubeconfig.sh
chmod +x ./create-multikueue-kubeconfig.sh
KUBECONFIG=worker-1.kubeconfig ./create-multikueue-kubeconfig.sh worker-1-multikueue.kubeconfig
KUBECONFIG=worker-2.kubeconfig ./create-multikueue-kubeconfig.sh worker-2-multikueue.kubeconfig

Install jobset api on worker cluster

KUBECONFIG=worker-1.kubeconfig kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.8.1/manifests.yaml
KUBECONFIG=worker-2.kubeconfig kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.8.1/manifests.yaml

Install Worker Kubeconfig in Manager Cluster

Store the worker cluster kubeconfig files as Kubernetes secrets within the kueue-system namespace on your manager cluster. This allows the MultiKueue controller to securely access them.

kubectl create secret generic worker-1-secret -n kueue-system --from-file=kubeconfig=worker-1-multikueue.kubeconfig
kubectl create secret generic worker-2-secret -n kueue-system --from-file=kubeconfig=worker-2-multikueue.kubeconfig

Configure MultiKueue Cluster on the Manager Cluster

Define MultiKueueCluster resources on the manager cluster. These resources link to the secrets containing the worker cluster kubeconfig files, allowing the manager to discover and manage the worker clusters.

kubectl apply -f - <<EOF
apiVersion: kueue.x-k8s.io/v1beta1
kind: MultiKueueCluster
metadata:
  name: multikueue-worker-1
spec:
  kubeConfig:
    locationType: Secret
    location: worker-1-secret

---
apiVersion: kueue.x-k8s.io/v1beta1
kind: MultiKueueCluster
metadata:
  name: multikueue-worker-2
spec:
  kubeConfig:
    locationType: Secret
    location: worker-2-secret
EOF

Verify MultiKueue Connectivity

kubectl get MultiKueueCluster

Apply Kueue config

On the worker clusters: https://github.com/maci0/multikueue/blob/main/kueue-config-worker.yaml On the manager clusters: https://github.com/maci0/multikueue/blob/main/kueue-config-manager.yaml