Works on AMD RX 7900 XT on Windows, but VRAM doesn't clear after each batch

[Willektron]

Running on a 7900 xt with 20 GB of VRAM.
Using no extra options (or using --medvram; doesn't make a difference to the eventual outcome).

Similar to the symptoms in the linked issue, when running multiple batches (either by setting the batch count to more than 1, or by simply running an individual batch one after another), the VRAM usage continues to increase instead of being cleared after each batch. So although I have 20GB of VRAM, while the first few batches run quite quickly, it eventually reaches a point where image generation slows down drastically (from ~5 iterations/second to ~1.3 seconds/iteration).

I'm guessing it's some kind of memory leak, but I don't know if it's related to the already solved issue in the original webui or not:
AUTOMATIC1111#1147

[Miraihi]

Yes it doesn't clear any garbage, that's the prevalent problem with this version of SD. But it's appears to be DirectML issue. We all have to wait until microsoft developers make improvements in vram usage.

[Willektron]

I was able to get it work at full speed consistently with "--opt-sub-quad-attention --no-half --precision full". Task manager shows VRAM getting maxed out after several batches but I don't receive errors and images seem to generate just fine; however I will do further testing and reply here again.

[Willektron]

Added --disable-nan-check for consistency

[saltyollpheist]

In the same boat you are - same GPU, although the command args you included here along with @Miraihi have helped immensely and make it so that I no longer have to close and relaunch. Will say though, you learn quickly not to necessarily trust the progress bar after a while. That being said, after about the 4th or 5th image, GPU finally crashed, but memory won't flush.

[saltyollpheist]

Ok, I'm still very new to all of this; so, take my input with a grain of salt. But I figured it worth noting some changes I've made that have made it so that I'm able to generate a plethora of images back to back without the need to restart. Again, no idea if any of this matters or not; so, I'm not excluding any changes I've made since my previous post. Pardon any "duh" moments and/or redundancy.

1. Open CMD in the root of the directory stable-diffusion-webui-directml.
2. git pull to ensure latest update
3. pip install -r requirements.txt <- it was at this point I knew I effed up during initial setup because I saw several missing items getting installed.
4. For the webui-user.bat file, I added the following line - set SAFETENSORS_FAST_GPU=1.

Restarted webui-user.bat and have been going just fine ever since. As always with these sorts of things, YMMV. Hope this helps in some way.

[Miraihi]

Update - I've been able to generate 800x1024 image with a speed of about 6,6 sec/it. Seems to be the current limit. Sampler - DPM++ 2M Karras.
Full arguments:

set COMMANDLINE_ARGS=--medvram --precision full --no-half --no-half-vae --opt-split-attention --opt-sub-quad-attention --disable-nan-check
set SAFETENSORS_FAST_GPU=1

[saltyollpheist]

@Miraihi
Followed your lead with those command args you listed using DPM++ 2M Karras.
A single batch of 4 count 512x512 in 50 seconds.
A single batch of 4 count 800x1024 in 15 min 24 seconds.

Not bad at all. I'll keep tweaking settings as I go and report back if I notice any differences or improvements regarding hirez fix.

[Miraihi]

Another update - the ControlNet extension works fine without lowvram flag if you use t2iadapter modules. Just don't forget to stick to 8-divisible image sizes and provide the correct .yaml alongside the model how you would do with .vae for pre-trained models.
I've used the newer coadapter-canny-sd15v1 (coadapter-sketch-sd15v1 is actually even better) together with the current sketch_adapter_v14.yaml provided by the ControlNet for a decent results (canny preprocessor for both). Setting weight to about 0.8 produces better images.

[Kademo15]

Hey I tried using sketch and it worked fine i then tried a depth-map because i wanted to recreate a video i saw online and i just get a gradient no actual depth-map. I followed your instructions to the point where you mentioned the .yaml file. I dont really know what that is or what I should do. Can you please elaborate on that.

[Miraihi]

Regarding yaml for the adapters - read the ControlNet readme file carefully, there is a part on the T2I adapters.
For the preprocessor use depth_leres instead of depth.

For depth model you need image_adapter_v14.yaml you can find in stable-diffusion-webui-directml\extensions\sd-webui-controlnet\models\. Copy and rename it so it's the same as the model (in your case coadapter-depth-sd15v1.yaml) and place alongside the model.

[aveeeteeeeee2]

Having the same issue.

[howDoILearnPython]

Have same issue here.... i'm using on board Vega 8 with 8GB dedicated VRAM.... after several round of image generation, OOM error will appear....

[lshqqytiger]

Because DirectML does not provide any way to clear VRAM, I searched and tried to empty it manually (and also memory-safely), but I couldn't find anything.

[pw405]

100% surefire method to clear VRAM without quitting web browser or closing the command prompt: let computer enter sleep, wake from sleep! Mostly joking, but this does actually work!... granted it is utterly impractical.
I'm using a 7900XTX and have been searching for VRAM over-use solutions. I figure these are still early days.

THANK YOU @lshqqytiger for the work you've done to bring SD to AMD GPU users! I would have never tried Stable Diffusion otherwise! Me and a few friends are loving it!

[lshqqytiger]

I think Olive will be better option for RX 7900 XTX users who are struggling with vram problems.
It has more restrictions on features (e.g. extra networks), but it clears memory automatically and is faster.

[pw405]

Forgive my ignorance... but I thought this was using Olive? I'm not super familiar with what all this means. I write tons of SQL for work, but only have a cursory understanding of everything in the development world.
(referring to post on AMD's GPUOpen here regarding Olive: https://gpuopen.com/amd-microsoft-directml-stable-diffusion/)

[lshqqytiger]

#149
You can optimize your model using Olive. (but it brings some restrictions on generation)
It uses ONNX runtime DmlExecutionProvider during generation which spends less vram and more efficient than torch-directml (non-olive way).

[Maverous]

Update - I've been able to generate 800x1024 image with a speed of about 6,6 sec/it. Seems to be the current limit. Sampler - DPM++ 2M Karras. Full arguments:

set COMMANDLINE_ARGS=--medvram --precision full --no-half --no-half-vae --opt-split-attention --opt-sub-quad-attention --disable-nan-check
set SAFETENSORS_FAST_GPU=1

I'm running on a RX 6600 XT. I know its not the same card but its AMD all the same. I never had VRAM issues on SD1.4 and just recently decided to pull the directml SD 1.5. Would constantly runout of VRAM after one or two image generations.

So far this has been working to keep my safetensors from going ape on my VRAM. I had no problems using SD 1.4, and upped to the 1.5 with directml. After adding this fix to my webui-user.bat, I've been able to avoid running out of VRAM (for now).
I'm still new to python scripting in general, but this fix has prevented my VRAM from maxing out after two image generations. That and disabling the browser accelerations. My bat file has the following commands in case this helps anyone:

 @echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS= --medvram --no-half --precision full --opt-sub-quad-attention --autolaunch --skip-torch-cuda-test
set SAFETENSORS_FAST_GPU=1

call webui.bat

[pw405]

wow, that's fast! What model are you using? You're getting over 6 Iterations/second at 800X1024 on a 6600XT? Just to clarify, you're not using ROCm on Linux?

[Kademo15]

@pw405 You got confused with it/sec and sec/it there. He gets 6,6sec/it.

[Miraihi]

Token Merging improves the performance massively (0.5 is a good value. The speedup in my case was more that 2 times faster), but you won't be able to use sub-quadratic optimization. DoggetX is a good alternative.

[Maverous]

wow, that's fast! What model are you using? You're getting over 6 Iterations/second at 800X1024 on a 6600XT? Just to clarify, you're not using ROCm on Linux?
Nope. Not getting the best results depending on the model now, but I am getting faster results than I was before running that. Before I was running on CPU only despite having all the arguments for my AMD card. Those Args have been allowing me to use pruned and smaller models much faster than torch 2.0.0+cpu that I was running on SD 1.4.

I want to get a 7900XT, but I'm thinking I might have to get away from my AMD gaming setup and dive into Nvidia cards soon for my sanity and to stop getting my VRAM overloaded.

[Miraihi]

To be honest, if you love Stable Diffusion/KoboldAI Nvidia is a must. I'm here with you, as much as I respect lshqqytiger's work.

[Miraihi]

AMD could've at least made ROCm more ubiqutous and make it work under Windows, but they didn't. The effort should go both ways - manufacturer's and consumer's. And the manufacturer clearly doesn't care enough at this point.

[pw405]

It seems there's a big push with current AI craze to get ROCm on Windows. I remember them making an official statement that it would have a release "This fall"... Hopefully they are right. They've published two articles on the Radeon blog specifically about Stable Diffusion, so there is at least some attention at AMD about SD.

I'm trying to get Olive/Onnnx working on my 7900XTX to test, but having tons of problems.

[pw405]

FWIW, using the 7900XTX has been pretty darn good for Stable Diffusion. VRAM fills, but I can generate dozens of SDXL images without crash or anything.

I understand your pain though, I'd like to be able to run everything natively in CUDA.

I wish Nvidia's VRAM choices weren't so strange. If you're OK with risking used, an old 3090 might be a solid choice. Granted, used cards from that era have a high chance of being a mining card.

Keep an eye for sales, I remember seeing a new 3090 for $650 or so about a month back.

[Maverous]

I'm not holding my breath. I can't get a Linux clone and my vram keeps getting clogged up on my 66,000 XT no matter what I do even with direct ml Sent from Yahoo Mail on Android On Mon, Aug 21, 2023 at 1:42 PM, ***@***.***> wrote: It seems there's a big push with current AI craze to get ROCm on Windows. I remember them making an official statement that it would have a release "This fall"... Hopefully they are right. They've published two articles on the Radeon blog specifically about Stable Diffusion, so there is at least some attention at AMD about SD. I'm trying to get Olive/Onnnx working on my 7900XTX to test, but having tons of problems. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>