Proper way use MULTI GPUs with DEEPSPEED when each GPU does not have enough VRAM?

[atipasvanund]

What's the proper way to fine-tune on MULTI GPUs with DEEPSPEED when each GPU does not have enough VRAM? For example If I wish to use 4 x RTX 4090 to fine-tune a 7B model without LORA or Quantization. Is this possible? I tried zero1.json all the way to zero3_bf16.json and get various error, including out of CUDA memory. Can someone please help me?

[NanoCode012]

You should be able to using zero3 with offload. Alternatively , make sure to adjust optim etc.