Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error running job: 'NoneType' object is not callable #243

Closed
D3voz opened this issue Jan 28, 2025 · 16 comments · May be fixed by #244
Closed

Error running job: 'NoneType' object is not callable #243

D3voz opened this issue Jan 28, 2025 · 16 comments · May be fixed by #244

Comments

@D3voz
Copy link

D3voz commented Jan 28, 2025 •
edited
Loading

This is for bugs only

Did you already ask in the discord?

Yes
You verified that this is a bug and not a feature request or question by asking in the discord?

Yes/No

Result:

  • 0 completed jobs
  • 1 failure
    ========================================
    Traceback (most recent call last):
    File "E:\ai-toolkit\run.py", line 90, in
    main()
    File "E:\ai-toolkit\run.py", line 86, in main
    raise e
    File "E:\ai-toolkit\run.py", line 78, in main
    job.run()
    File "E:\ai-toolkit\jobs\ExtensionJob.py", line 22, in run
    process.run()
    File "E:\ai-toolkit\jobs\process\BaseSDTrainProcess.py", line 1853, in run
    loss_dict = self.hook_train_loop(batch_list)
    File "E:\ai-toolkit\extensions_built_in\sd_trainer\SDTrainer.py", line 1659, in hook_train_loop
    loss = self.train_single_accumulation(batch)
    File "E:\ai-toolkit\extensions_built_in\sd_trainer\SDTrainer.py", line 1606, in train_single_accumulation
    noise_pred = self.predict_noise(
    File "E:\ai-toolkit\extensions_built_in\sd_trainer\SDTrainer.py", line 968, in predict_noise
    return self.sd.predict_noise(
    File "E:\ai-toolkit\toolkit\stable_diffusion_model.py", line 1871, in predict_noise
    noise_pred = self.unet(
    File "E:\ai-toolkit\venv\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
    File "E:\ai-toolkit\venv\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
    File "E:\ai-toolkit\venv\lib\site-packages\diffusers\models\transformers\transformer_flux.py", line 520, in forward
    encoder_hidden_states, hidden_states = self._gradient_checkpointing_func(
    TypeError: 'NoneType' object is not callable
    my_first_flux_lora_v1: 0%| | 0/2000 [00:00<?, ?it/s]

The error comes from turning gradient checkpointing , on windows .

Turning Gradient checkpointing off makes it go out of memory - Caching latents to disk: 100%|███████████████████████████████████████████████████████████████████| 1/1 [00:00<?, ?it/s]
Generating baseline samples before training
my_first_flux_lora_v1: 0%| | 0/2000 [00:01<?, ?it/s, lr: 1.0e-04 loss: 5.966e-01]Error running job: CUDA out of memory. Tried to allocate 216.00 MiB. GPU 0 has a total capacity of 79.54 GiB of which 120.81 MiB is free. Of the allocated memory 70.52 GiB is allocated by PyTorch, and 1.05 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

========================================
Result:

  • 0 completed jobs
  • 1 failure
    ========================================
@D3voz
Copy link
Author

D3voz commented Jan 28, 2025

Fixed it, by uninstalling diffuser and reinstalling it and going back to an older commit

@D3voz D3voz closed this as completed Jan 28, 2025
@jhj0517 jhj0517 mentioned this issue Jan 28, 2025
Open
@jhj0517
Copy link

jhj0517 commented Jan 28, 2025 •
edited
Loading

@D3voz Can you keep it open until it is actually fixed in the repository?

+) I made #244 for it

@jhj0517 jhj0517 mentioned this issue Jan 28, 2025
Closed
@AfterHAL
Copy link

AfterHAL commented Jan 28, 2025 •
edited
Loading

I got this error too.
Uninstalled "diffusers" with pip uninstall diffusers
-> Found existing installation: diffusers 0.33.0.dev0
-> Uninstalling diffusers-0.33.0.dev0:

and reinstalled with pip install diffusers
-> Installing collected packages: diffusers
-> Successfully installed diffusers-0.32.2

@D3voz
Copy link
Author

D3voz commented Jan 28, 2025

I think the issue is caused by diffusers-0.33.0.dev0 . Reinstalling diffuser with pip install diffusers fixes it.

@D3voz D3voz reopened this Jan 28, 2025
@D3voz
Copy link
Author

D3voz commented Jan 28, 2025

@D3voz Can you keep it open until it is actually fixed in the repository?

ok

@theprasunbanerjee
Copy link

I uninstalled and reinstalled it yet showing the same thing what to do , am using AI toolkit on runpod till yesterday everythin was alright idk what happened today

@SAC020
Copy link

SAC020 commented Jan 29, 2025

It's still not working in Colab

@goffy76
Copy link

goffy76 commented Jan 29, 2025

I got this error too. Uninstalled "diffusers" with pip uninstall diffusers -> Found existing installation: diffusers 0.33.0.dev0 -> Uninstalling diffusers-0.33.0.dev0:

and reinstalled with pip install diffusers -> Installing collected packages: diffusers -> Successfully installed diffusers-0.32.2

Works - thank you hero :-)

@SAC020
Copy link

SAC020 commented Jan 30, 2025

I think the issue is caused by diffusers-0.33.0.dev0 . Reinstalling diffuser with pip install diffusers fixes it.

workaround worked for me as well, but what needs to happen for the issue to be fixed at the source?

@jhj0517
Copy link

jhj0517 commented Jan 30, 2025

@SAC020 ai-toolkit installs diffusers from the source code, not from pypi (I guess to follow recent patches from diffusers), and some recent patches from diffusers caused incompatibility with ai-toolkit.

To fix this from source you can see #244

@SAC020
Copy link

SAC020 commented Jan 30, 2025

@SAC020 ai-toolkit installs diffusers from the source code, not from pypi (I guess to follow recent patches from diffusers), and some recent patches from diffusers caused incompatibility with ai-toolkit.

To fix this from source you can see #244

Thank you for explaining and for the fix. I am not sure how will I know / how can I tell when the proposed change is merged into the main code, I am not very proficient with github

@eliohead
Copy link

eliohead commented Feb 2, 2025 •
edited
Loading

In the Colab notebook, this issue is resolved by uninstalling and reinstalling diffusers. However, since this causes the session to restart and the variables to be lost, you should not restart running the first cell, otherwise, the old non-functional diffusers will be reinstalled again. Instead, restart from the cell where the HF tokens are set.

@khurchla
Copy link

khurchla commented Feb 5, 2025

Downgrading to diffusers Version: 0.32.2 also resolved this error for me on RunPod today. I uninstalled and reinstalled it with pip from PyPI as others recommended. Thanks very much.

@Xhah2017
Copy link

Xhah2017 commented Feb 7, 2025

I got this error too.
Uninstalled "diffusers" with pip uninstall diffusers
-> Found existing installation: diffusers 0.33.0.dev0
-> Uninstalling diffusers-0.33.0.dev0:

and reinstalled with pip install diffusers
-> Installing collected packages: diffusers
-> Successfully installed diffusers-0.32.2

@AfterHAL Thank you, saved me twice.

@SAC020
Copy link

SAC020 commented Feb 9, 2025 •
edited
Loading

I think it's been fixed at the source, out-of-the-box now it installs diffusers 0.32.2

0e75724

@D3voz
Copy link
Author

D3voz commented Feb 10, 2025

Fixed

@D3voz D3voz closed this as completed Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use enable_gradient_checkpointing() for flux jhj0517/ai-toolkit
9 participants
@khurchla @Xhah2017 @AfterHAL @SAC020 @eliohead @theprasunbanerjee @jhj0517 @D3voz @goffy76