Add Support for Managed Instance Groups to Enable Dynamic Workload Scheduler

[jhamet93]

Dynamic Workload Scheduler is a resource management and job scheduling platform currently integrated into different Google product surface areas to improve access to hard to obtain accelerators. At the Compute Engine surface area, you create a Managed Instance Group (MIG) with a resize request detailing how long you need the VM's for. If possible then, the job duration configured via sbatch could then be used to detail how long the VM's are needed for.

Is there an appetite or opportunity to support utilizing Dynamic Workload scheduler for dynamic scaling up of nodes for hard to obtain accelelerators ? Would be more than happy to contribute if so as this would be a boon to our Cloud Slurm Cluster.

[cboneti]

Hey Josh,

We plan on supporting DWS this year and are exploring various alternatives. You mentioned we could use the job duration for sbatch, which is in line with what we also thought. I was wondering if you could describe a bit more what you would like to see in terms of functionality or user experience? For instance, what machine types would be more interesting for you? Are you thinking about mostly "exclusive" nodes that are created for a single job, or would you like to have a partition with "semi static nodes" that serve multiple jobs? Please tell us as much as you would feel comfortable here. Thanks!

[casassg]

Hi @cboneti (same team as Josh here), I think for our use case probs both would be interesting:

• DWS to get some a100 nodes (or h100) for training ML Models regularly (aka we need to retrain model A which requires us to have a node w 16 A100 GPUs every week, DWS can allow us to get more easily provisioned GPU nodes for the duration of the job).
• Semi-static also is interesting specially if DWS has better luck at finding GPU nodes. Right now the biggest issue we find is that we frequently get stockout for ephemeral jobs (aka modelers experimenting w new model architecture and such). So semi-static would make sense (even userExclusive allowing multiple jobs to be provisioned without tearing down if it fails for fast iteration).

[cboneti]

Yes, we want to enable DWS as soon as possible.
From what I hear, you would like to create the nodes with DWS for a short period of time. I was wondering if you wanted to create a node per job (currently, the exclusive flag in our slurm partitions), or if you wanted to create node-set where nodes were provisioned for the full 7 days and then multiple jobs were streamed through this.

We hear you and we are working on it but we don't have an ETA just yet.

[casassg]

or if you wanted to create node-set where nodes were provisioned for the full 7 days and then multiple jobs were streamed through this.

I think it would be interesting though I assume integrating then w calendar reservations may make more sense? Would be curious what the experience would look like in the case of 7 days. At the moment our current cluster sits idle until its working hours when modeling teams use to train some models, so we are mostly trying to figure out best way we can do to garantee their access to GPUs or for future when we have scheduled retraining to make sure a certain time of day a model has available nodes to train on for a specific time period (aka indeed exclusive flag)

[casassg]

hi @mr0re1 wanted to check in wether there was any progress or any update on this :D

[mr0re1]

Hi @casassg
The trivial support for MIG backed instances will be added by GoogleCloudPlatform/cluster-toolkit#2696 (ETA early next week).
It uses dynamic/configless setup, so Slurm scheduler will not attempt to suspend/resume any nodes in this nodeset.
It enables wide range of instance management approaches, as long as they are done outside of Slurm )

Please let us know if it is any help for you, report any bugs you will find.

As for the more tight integration with MIG we are working on it, but I can't provide any details, time.

[casassg]

@mr0re1 been following along, current implementation though doesn't use DWS right? As that requires backend to issue resize request (or it can but requires you to request them from the UI)

[mr0re1]

@casassg this implementation doesn't do ANY "resizing" efforts on Slurm side. If you want to use DWS , you can do it manually, nodes will register with Slurm automatically. E.g. deploy blueprint with 0-sized MIG; issue resize request backed by DWS manually agains this MIG.

This is not our "intended user experience with MIG", but rather an "improved support for dynamic/configless nodes"