Speed of image generation and max. resolution

[aidik]

Hi, I would like to find out, what I'm doing wrong.

I'm currently using the latest 0.9 version of basujindal/stable-diffusion code with weights sd-v1-4-full-ema.ckpt sourced from Hugging Face on a system with 3090 Ti, i7-6700, 64 GB of RAM, and a Consumer SSD.

If I run:

(stable-diffusion) C:\Users\Aidik\Documents\Python Scripts\stable-diffusionOPT>python optimizedSD/optimized_txt2img.py --prompt "Cyberpunk style image of a Telsa car reflection in rain" --H 512 --W 512 --seed 27 --n_iter 1 --n_samples 1 --ddim_steps 50

It takes about 1min and 55 seconds to finish.

Global seed set to 27
Loading model from models/ldm/stable-diffusion-v1/model.ckpt
Global Step: 470000
UNet: Running in eps-prediction mode
CondStage: Running in eps-prediction mode
FirstStage: Running in eps-prediction mode
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Sampling:   0%|                                                                                  | 0/1 [00:00<?, ?it/s]seeds used =  [27]                                                                                | 0/1 [00:00<?, ?it/s]
Data shape for PLMS sampling is [1, 4, 64, 64]
Running PLMS Sampling with 50 timesteps
PLMS Sampler: 100%|████████████████████████████████████████████████████████████████████| 50/50 [01:11<00:00,  1.44s/it]
torch.Size([1, 4, 64, 64])
saving images 100%|████████████████████████████████████████████████████████████████████| 50/50 [01:11<00:00,  1.36s/it]
memory_final =  2.575872
data: 100%|██████████████████████████████████████████████████████████████████████████████| 1/1 [01:17<00:00, 77.09s/it]
Sampling: 100%|██████████████████████████████████████████████████████████████████████████| 1/1 [01:17<00:00, 77.09s/it]
Samples finished in 1.77 minutes and exported to outputs/txt2img-samples\Cyberpunk_style_image_of_a_Telsa_car_reflection_in_rain
 Seeds used = 27

If I try to run:

(stable-diffusion) C:\Users\Aidik\Documents\Python Scripts\stable-diffusionOPT>python optimizedSD/optimized_img2img.py --prompt "Austrian alps" --init-img input/alps.png --strength 0.8 --n_iter 1 --n_samples 1 --H 1088 --W 1152

Which according to the readme file should run on 4× less memory (6GB GPU (RTX 2060)) I get the following out-of-memory error:

Global seed set to 303987
Loading model from models/ldm/stable-diffusion-v1/model.ckpt
Global Step: 470000
loaded input image of size (512, 512) from input/alps.png
New image size (1152, 1088)
optimizedSD/optimized_img2img.py:49: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  image = image.resize((w, h), resample=Image.LANCZOS)
UNet: Running in eps-prediction mode
CondStage: Running in eps-prediction mode
FirstStage: Running in eps-prediction mode
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
target t_enc is 40 steps
Sampling:   0%|                                                                                  | 0/1 [00:00<?, ?it/s]seeds used =  [303987]                                                                            | 0/1 [00:00<?, ?it/s]
Running DDIM Sampling with 40 timesteps
Decoding image:   0%|                                                                           | 0/40 [00:04<?, ?it/s]
data:   0%|                                                                                      | 0/1 [00:04<?, ?it/s]
Sampling:   0%|                                                                                  | 0/1 [00:04<?, ?it/s]
Traceback (most recent call last):
  File "optimizedSD/optimized_img2img.py", line 319, in <module>
    samples_ddim = model.sample(
  File "C:\Users\Aidik\anaconda3\envs\stable-diffusion\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\optimizedSD\ddpm.py", line 529, in sample
    samples = self.ddim_sampling(x_latent, conditioning, S, unconditional_guidance_scale=unconditional_guidance_scale,
  File "C:\Users\Aidik\anaconda3\envs\stable-diffusion\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\optimizedSD\ddpm.py", line 710, in ddim_sampling
    x_dec = self.p_sample_ddim(x_dec, cond, ts, index=index, use_original_steps=use_original_steps,
  File "C:\Users\Aidik\anaconda3\envs\stable-diffusion\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\optimizedSD\ddpm.py", line 732, in p_sample_ddim
    e_t_uncond, e_t = self.apply_model(x_in, t_in, c_in).chunk(2)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\optimizedSD\ddpm.py", line 402, in apply_model
    h,emb,hs = self.model1(x_noisy[0:step], t[:step], cond[:step])
  File "C:\Users\Aidik\anaconda3\envs\stable-diffusion\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\optimizedSD\ddpm.py", line 312, in forward
    out = self.diffusion_model(x, t, context=cc)
  File "C:\Users\Aidik\anaconda3\envs\stable-diffusion\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\optimizedSD\openaimodelSplit.py", line 608, in forward
    h = module(h, emb, context)
  File "C:\Users\Aidik\anaconda3\envs\stable-diffusion\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\optimizedSD\openaimodelSplit.py", line 72, in forward
    x = layer(x, context)
  File "C:\Users\Aidik\anaconda3\envs\stable-diffusion\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\ldm\modules\attention.py", line 258, in forward
    x = block(x, context=context)
  File "C:\Users\Aidik\anaconda3\envs\stable-diffusion\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\ldm\modules\attention.py", line 209, in forward
    return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\ldm\modules\diffusionmodules\util.py", line 114, in checkpoint
    return CheckpointFunction.apply(func, len(inputs), *args)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\ldm\modules\diffusionmodules\util.py", line 127, in forward
    output_tensors = ctx.run_function(*ctx.input_tensors)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\ldm\modules\attention.py", line 212, in _forward
    x = self.attn1(self.norm1(x)) + x
  File "C:\Users\Aidik\anaconda3\envs\stable-diffusion\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "c:\users\aidik\documents\python scripts\stable-diffusion\ldm\modules\attention.py", line 191, in forward
    out = einsum('b i j, b j d -> b i d', attn, v)
  File "C:\Users\Aidik\anaconda3\envs\stable-diffusion\lib\site-packages\torch\functional.py", line 330, in einsum
    return _VF.einsum(equation, operands)  # type: ignore[attr-defined]
RuntimeError: CUDA out of memory. Tried to allocate 5.72 GiB (GPU 0; 23.99 GiB total capacity; 17.90 GiB already allocated; 3.06 GiB free; 17.99 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Clearly, I'm doing something wrong, but what? Do I use the wrong weights? Do I miss some environment variables?
Thank you for your suggestion and help.

P.S: I'm not using the issue feature because I believe there is an issue with my setup, not the code. :-)

[bitRAKE]

I'd try to re-install. (Maybe something else is using the GPU memory?)

I'm on Win11 w/ 1080ti, and I just did:
(from a conda prompt with git installed)

git clone <repo> test
cd test
copy ..\sd-v1-4.ckpt models/ldm/stable-diffusion-v1/model.ckpt
conda env create -f environment.yaml
conda activate ldm
python optimizedSD\optimized_img2img.py --skip_grid --from-file "..\prompts.txt" --init-img "..\init.png" --strength 0.79 --outdir ".." --W 1280 --H 1280 --ddim_steps 40 --scale 7.5 --n_samples 1

... and Task Manager is showing between 8.8GB and 10.4GB of GPU memory use. Each image takes about 4:30, but that is because 1080ti's don't have uber FP16 speed.

[aidik]

Thank you, this worked. I started with the smaller 4GB model and got to about 10.5GB of GPU RAM used while generating. It took about 2:30 to gen the image.