Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug]: Update to 4.2.4 causes CUDA out of memory error in WSL/docker #6542

Open
1 task done
jozzo402 opened this issue Jun 25, 2024 · 1 comment
Open
1 task done
Labels
bug Something isn't working

Comments

@jozzo402
Copy link

Is there an existing issue for this problem?

  • I have searched the existing issues

Operating system

Linux

GPU vendor

Nvidia (CUDA)

GPU model

RTX 2060 Super

GPU VRAM

8GB

Version number

4.2.4

Browser

Waterfox G6.0.16

Python dependencies

No response

What happened

When generating with many LORAs added, Invoke 4.2.4 reports that my system is out of memory:
image

It seems to be filling VRAM, but doesn't fall back to system RAM once VRAM is full, then crashes.
I've changed the fallback setting in Nvidia console to system, it makes no difference.
If I use Invoke 4.2.1, it works fine. Unsure about versions in between.

What you expected to happen

Overflowing VRAM should fallback to system RAM, and overflowing system RAM should fallback to Swap. It should not throw an error once VRAM is full.

How to reproduce the problem

Using my setup install Invoke 4.2.1 and generate an image with heavy memory requirements (SDXL, many loras, controlnets, etc). It will work fine.
Then, update to Invoke 4.2.4 and generate another image with the same parameters. It will no longer work and you'll see this error. (use random seed, otherwise it might just copy the last image)

Additional context

My setup is Windows10 > WSL2 > Debian12 > Docker > InvokeAI. This is my docker compose:

invokeai-nvidia:
  build:
    context: ./InvokeAI
    dockerfile: docker/Dockerfile
  environment:
    - INVOKEAI_ROOT=/invokeai
    - INVOKEAI_LOG_LEVEL=info
    - INVOKEAI_HASHING_ALGORITHM=blake3_multi
  volumes:
    - ./invokeai:/invokeai
    - ./models:/invokeai/models
    - /mnt/g/Main/VirtualBox/WSL/models:/invokeai/models/g-mnt
  network_mode: host
  tty: true
  stdin_open: true
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]

Discord username

No response

@jozzo402 jozzo402 added the bug Something isn't working label Jun 25, 2024
@jozzo402
Copy link
Author

jozzo402 commented Jun 26, 2024

Update:
Have just finished testing older versions, is definitely an issue specifically with 4.2.4.
Everything up to 4.2.3 works fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant