Unable to run benchmark #170
Description
I'm trying to run the txt2img benchmark, but can't get it to work.
I tried several combinations of versions for deepspeed (incl. from source), deepspeed-mii (incl. from source), torch==1.13.1 (also with 1.12.1 when using cuda 11.6), diffusers, transformers and triton.
Environment:
- OS: Amazon Linux 2 w/kernel 5.10
- GPU: Nvidia Tesla T4
- python version:
3.9, compiled from source - nvidia driver:
510.108.03-grid - cuda version:
11.7
pip install -U deepspeed==0.7.5 deepspeed-mii==0.0.3 torch==1.13.1 diffusers==0.7.1 transformers==4.24.0 triton==2.0.0.dev20221030 ftfy, pip install -U git+https://github.com/microsoft/DeepSpeed.git@c9c6ab9e32b054136c3a125900d6e2ed937432be deepspeed-mii==0.0.4 torch==1.13.1 diffusers==0.11.1 transformers==4.24.0 triton==2.0.0.dev20221202 ftfy, pip install -U deepspeed==0.8.0 deepspeed-mii==0.0.4 torch==1.13.1 diffusers==0.11.1 transformers==4.24.0 triton==2.0.0.dev20221202 ftfy, pip install -U deepspeed==0.8.0 deepspeed-mii==0.0.4 torch==1.13.1 diffusers==0.11.1 transformers==4.24.0 triton==2.0.0.dev20221202 ftfy, pip install -U deepspeed==0.7.7 deepspeed-mii==0.0.4 torch==1.13.1 diffusers==0.10.2 transformers==4.24.0 triton==2.0.0.dev20221202 ftfy all fail with {created_time:"2023-04-24T15:34:23.563287825+00:00", grpc_status:2, grpc_message:"Exception calling application: Triton Error [CUDA]: invalid argument"}"
pip install -U deepspeed==0.9.1 deepspeed-mii==0.0.3 torch==1.13.1 diffusers==0.14.0 transformers==4.24.0 triton==2.0.0.dev20221202 ftfy fails with ImportError: cannot import name 'CLIPTextModelWithProjection' from 'transformers' (/home/ec2-user/.local/lib/python3.9/site-packages/transformers/__init__.py)
pip install -U deepspeed==0.9.1 deepspeed-mii==0.0.3 torch==1.13.1 diffusers==0.14.0 transformers==4.26.0 triton==2.0.0.dev20221202 ftfy fails with AttributeError: 'StableDiffusionPipeline' object has no attribute 'children'
pip install -U deepspeed==0.7.5 deepspeed-mii==0.0.3 torch==1.13.1 diffusers==0.11.1 transformers==4.24.0 triton==2.0.0.dev20221202 ftfy fails with {created_time:"2023-04-24T16:19:36.666852314+00:00", grpc_status:2, grpc_message:"Exception calling application: \'DSUNet\' object has no attribute \'config\'"}"
pip install -U git+https://github.com/microsoft/DeepSpeed.git@35eabb0a336e7a8e9950a550475ceaebda42066c deepspeed-mii==0.0.4 torch==1.13.1 diffusers==0.11.1 transformers==4.24.0 triton==2.0.0.dev20221202 ftfy fails with {grpc_message:"Exception calling application: forward() got an unexpected keyword argument \'encoder_hidden_states\'", grpc_status:2, created_time:"2023-04-24T16:34:55.067998104+00:00"}". Same for pip install -U deepspeed==0.7.7 deepspeed-mii==0.0.4 torch==1.13.1 diffusers==0.11.1 transformers==4.24.0 triton==2.0.0.dev20221202 ftfy
Where this issue seems relevant.
pip install -U deepspeed==0.7.6 deepspeed-mii==0.0.4 torch==1.13.1 diffusers==0.11.1 transformers==4.24.0 triton==2.0.0.dev20221202 ftfy fails with {created_time:"2023-04-24T17:29:38.723537173+00:00", grpc_status:2, grpc_message:"Exception calling application: \'DSUNet\' object has no attribute \'config\'"}". Same for pip install -U deepspeed==0.7.6 deepspeed-mii==0.0.4 torch==1.13.1 diffusers==0.11.1 transformers==4.24.0 triton==2.0.0.dev20221202 ftfy
Where this other issue seems relevant.
Is there any set of versions that is well known to work?