Does deepspeed partition the model to multi GPUs?

[vikki7777]

I am trying to run your code with the flan-t5-base model, I have one machine with multi V100, each gpu has 16G memory, and then I meet the following question:
when I use single gpu, the gpu memory usage is 11.5G,
when I ues 4 gpus, each gpu memory usage is 11.7G,
deepspeed said it could partition the model, so I don't understand why gpu memory usage is almost same?

single gpu:

deepspeed --num_gpus=1  run_seq2seq_deepspeed.py 
    --model_id model 
    --dataset_path dataset 
    --epochs 3 
    --per_device_train_batch_size 16 
    --per_device_eval_batch_size 16 
    --generation_max_length 111 
    --lr 1e-4 
    --deepspeed configs/ds_flan_t5_z3_offload.json 

multi gpus:

deepspeed --num_gpus=4  run_seq2seq_deepspeed.py 
    --model_id model 
    --dataset_path dataset 
    --epochs 3 
    --per_device_train_batch_size 16 
    --per_device_eval_batch_size 16 
    --generation_max_length 111 
    --lr 1e-4 
    --deepspeed configs/ds_flan_t5_z3_offload.json 

ds_config:
{
"fp16": {
"enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1
},
"optimizer": {
"type": "AdamW",
"params": {
"lr": "auto",
"betas": "auto",
"eps": "auto",
"weight_decay": "auto"
}
},
"scheduler": {
"type": "WarmupLR",
"params": {
"warmup_min_lr": "auto",
"warmup_max_lr": "auto",
"warmup_num_steps": "auto"
}
},
"zero_optimization": {
"stage": 3,
"offload_optimizer": {
"device": "cpu",
"pin_memory": true
},
"offload_param": {
"device": "cpu",
"pin_memory": true
},
"overlap_comm": true,
"contiguous_gradients": true,
"sub_group_size": 1e9,
"reduce_bucket_size": "auto",
"stage3_prefetch_bucket_size": "auto",
"stage3_param_persistence_threshold": "auto",
"stage3_max_live_parameters": 1e9,
"stage3_max_reuse_distance": 1e9,
"stage3_gather_16bit_weights_on_model_save": true
},
"gradient_accumulation_steps": "auto",
"gradient_clipping": "auto",
"steps_per_print": 2000,
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": "auto",
"wall_clock_breakdown": false
}

[philschmid]

Your model is small enough to fit on a single GPU. Deepspeed then does Data parallelism and runs more models. You should see a faster time to train

[vikki7777]

I changed the other two models: flan-t5-xl and flan-t5-xxl.

flan-t5-xl with single gpu:

deepspeed --include localhost:1  run_seq2seq_deepspeed.py 
    --model_id model_xl 
    --dataset_path dataset 
    --epochs 3 
    --per_device_train_batch_size 1 
    --per_device_eval_batch_size 1 
    --generation_max_length 111 
    --lr 1e-4 
    --deepspeed configs/ds_flan_t5_z3_offload.json 

flan-t5-xl with six gpus:

deepspeed --include localhost:1,2,3,4,5,6  run_seq2seq_deepspeed.py 
    --model_id model_xl 
    --dataset_path dataset 
    --epochs 3 
    --per_device_train_batch_size 1 
    --per_device_eval_batch_size 1 
    --generation_max_length 111 
    --lr 1e-4 
    --deepspeed configs/ds_flan_t5_z3_offload.json 

flan-t5-xxl with single gpu:

flan-t5-xxl with six gpus: also cuda out of memory

For flan-t5-xxl, the results are both out of memory, the process of model partition cannot be seen；
For flan-t5-xl, I thought that when using a single gpu, if the memory occupied by the model is N, and then when using multiple gpus, the memory occupied by each gpu should be 1/N. Is this correct?
However, the results show that when using multiple gpus, the memory usage of each GPU increases (from 8G to 12G), why?
Looking forward to your reply :)

[vikki7777]

@philschmid Could you help me solve the above problem? Thanks a lot.

[yulinliu101]

Hi Vikki, did you figure out this issue? I had 4 V100s and have observed similar situations for the GPU memory management. Even with cpu offload, I wasn't able to fine-tune the Flan-T5-XL (fp32) on my hardware. Although it is also mysterious that I could tune it when loading the model with torch_dtype=torch.bfloat16,, which the OP said V100 does not support bf16 dtypes. Any insights or recommendations will be super helpful!

Thanks!

Attaching some dependencies info:

Cuda driver==12.1
transformers==4.35.2
torch==2.1.1+cu121
accelerate==0.25.0
peft==0.7.0
deepspeed==0.12.4