Forge Is Not Using ComfyUI as A Backend #169
Replies: 18 comments 49 replies
-
|
Below is another similar test with https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#alternating-words WebUI alternating-words allow for switching words each timestep like
Prompt [cow|cow|horse|man|siberian tiger|ox|man] in a field Negative bad, ugly WebUI Forge (webui install is on xformers, and forge is on sdp) Again, it is possible to do similar things in ComfyUI using nodes, but getting this same images is very difficult |
Beta Was this translation helpful? Give feedback.
-
|
Also like https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#composable-diffusion
This is webui
This is forge
a cat AND a dog |
Beta Was this translation helpful? Give feedback.
-
|
Ending note: You may find You may find many logics similar to comfyui codes (they do not have a PYPI package to import), but same with Finally, we reserve the right to defend our work against any misinformation in a more formal way if necessary. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
It feels like people are arguing over the usage of "uses ComfyUI's code in backend" vs "uses ComfyUI as backend" and the 1st one is more accurate than the 2nd from what I'm seeing |
Beta Was this translation helpful? Give feedback.
-
|
Kindergarten, yay! I'll see myself out... |
Beta Was this translation helpful? Give feedback.
-
|
Just as a non coder consumer parasite of you alls products, of which I love and treasure and thank you both so much, this is what it looked like to me. Maybe my ignorance in this issue will allow some forest for the trees insight. If not ill stfu. My impression of this repo which I was downloading is that ample credit was given to comfyui in fact I preceived it as essential and without it this repo woudl not exist. I only found this discussion while digging around waiting for the zip to extract. I have yet to fire it up and have no bias here if anything I am super comfy w/ comfy. But I never once thought I was downloading some unique completely original snowflake of a program designed from ground up. If that does anything for anyone. Two thoughts: #2: You both are saying the same thing just have different interpretations. It may be more productive to to come to that understanding to begin with. Whether correct or incorrect its clear that there are two beliefs in this matter so this must be accepted first before attempting to prove is false etc. Without understanding the perspective of this repo it will seem like malicious intent. Is crucial to focus on this to ever reach conclusion. For example, what would a distro look like vs this one that meets the mark of utilizing comfyui as backend from this one's perspective. In what amount of code does it differ. Then if you both can at least be starting from the same place of understanding you'll squash this easy and something like asking how the attribution should read to make happy would make piece of cake. I highly doubt anyone was out to steal anything. This repo likely thinks their implementation is unique even if sharing code similar to the examples picked from comfy code. probably i reget posting this just 2 cents of someone on outside |
Beta Was this translation helpful? Give feedback.
-
|
very interesting |
Beta Was this translation helpful? Give feedback.
-
|
aren't they all GNU v3 license, as long as you credited it. it's fine? |
Beta Was this translation helpful? Give feedback.
-
|
It is very clear this is still A1111 as the backend, with additional properly attributed code. It's not just A1111's UI plopped on top of another backend like the statement "using ComfyUI as a backend" misleadingly implies. I definitely trust the creator of ControlNet far more than a dev who still has a petty grudge against another project since 2022, and wastes all his time posting about it on certain vietnamese basket weaving sites, it especially now that they're a SAI employee. |
Beta Was this translation helpful? Give feedback.
-
|
In my opinion, the better statement is: Forge is not using Comfy as backend. And in my view, some of addon modules are just copied from comfy with some modification. I'm personally rooting for @comfyanonymous. |
Beta Was this translation helpful? Give feedback.
-
|
i think more so the issue is stating that this is based on auto's code, which it is mostly not (there are some parts i'd believe are from auto, though just a few). if illya just said that "it has some parts of [xyz]", this couldve been prevented |
Beta Was this translation helpful? Give feedback.
-
|
@FizzleDorf Note that this post has clearly said that
But some keep ignoring this and try to change my claims into something like 'forge has no relationship to comfyui' and then attack things that I never claimed.
I am willing to remove the discussion or post any declarations if Comfyanonymous is willing to apologize to diffusers team, and original Automatic1111 team, for the bad words that he/she said in public many times. |
Beta Was this translation helpful? Give feedback.
-
the issue here is:
when there are parts of the code that are in use, not just referenced. please stop being so abrasive. github is for collaboration, learning and understanding.
what does this have to do with the discussion? personal disputes should be handled by the parties involved and privately if there is actually any issues. making a declaration like this is exactly the behaviour you should not be expressing on github. |
Beta Was this translation helpful? Give feedback.
-
Forge is based on auto's code and mainly use Automatic1111 backend. Note that Automatic1111 like monkeypatch very much, and those codes are patched in runtime and you do not find them in place in static codes by reading codes in git. https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/sd_hijack.py So if you just review static codes in git, you will underestimate the amount of Automatic1111 engine |
Beta Was this translation helpful? Give feedback.
-
|
I don't think he mis-communicated at all and was exasperating himself to be clear on the issue. While others, cough-you-cough, were acting like they were flava flav in comfyanonymous' rap group. (hypeman). People were clearly loading their biases on this topic and following Comfy's lead. It's very disappointing to see such manufactured drama pulled deep from the annals of egos. The upper brass at stability should examine this situation close. Comfy claiming they refused to credit when he never did that. Language like "Straight up copied".. bruh it's open source. Emad needs to muzzle his pup. Peanut gallery all like "i'm rooting for"... i'm just sad to see open source communities fall into a steaming pile of social media dramarama. |
Beta Was this translation helpful? Give feedback.
-
|
Given the complexity of development here and nuances of open source contributions, it seems clear (to me at least) that there might be some misunderstandings happening. The presence of similar or the copied code from ldm doesn't necessarily imply direct copying of the backend, especially if certain parts of the code aren't currently in use. It could indicate that there was planning for future comfyui functionalities, or experimenting with different approaches. If that's true it probably should have been communicated since it's still present, but also understanding the intent behind decisions to add the unused comfyui ldm in the first place seems to require deeper insight into llyasviel's development process, something that isn't totally apparent from the outside. I think we should consider the possibility that what we're seeing a part of a larger and evolving strategy, which basically reflects the dynamic nature of coding and project evolution. But also, I could be totally wrong here. |
Beta Was this translation helpful? Give feedback.
-
|
I am not interested in any of your quarrel. But as someone who can code, I would clarify the following This software can be devided into 4 parts:
So what's my point?
Finally, to answer the question, Is Forge using ComfyUI as a backend? Thanks for your reading, |
Beta Was this translation helpful? Give feedback.
-
|
Totally agree. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, sad to hear some people say "it's better them automatic in everything", but the truth it is the automacitc's ui with a few modifications Maybe people haven't used sd-webui Dev branch and think all of these visual modifications are made by @lllyasviel 😕 |
Beta Was this translation helpful? Give feedback.
-
@light-and-ray I am willing to transfer the repo to any leadership of original webui team and maintain codes there. just let me know anytime |
Beta Was this translation helpful? Give feedback.
-
To avoid people misunderstanding and attacking things that we have never claimed/done, I am giving this FAQ as a final response.This is the last time that I respond to this thread (unless someone comes up with an argument that may deserve more considerations).Q: Do you have comfyui codes in Forge?Yes. And this has been made very clear in the first post. But this does not mean we are using comfyui backend. Q: Do you have diffusers codes in Forge?Yes. But this does not mean we are using diffusers backend. Q: If you do not use them as backend, why you import them?Because we use them to improve other backend (Automatic1111 backend) and we need to import them. If a project has If a project does not have pip or if the pip is difficult, we directly include codes, examples are Q: Then what is your backend?Automatic1111 Stable Diffusion WebUI backend. And this post has all examples to show the results. Q: Do you use comfyui codes?Yes. And this has been made very clear in the first post. We use a memory management system very similar to comfyui to improve Automatic1111, with several improvements to make it work better for webui. We also build a UNet patcher system that is similar but different to comfyui. The Forge UNet patcher system solves several problems for A1111 and A1111’s controlnet, like We also use comfyui’s many optimization methods to improve the webui inference, like more compact attention codes, etc. Q: Are you using comfyui to sample images?No. Our backend is Automatic1111 and we use Automatic1111 to sample images. Q: Are you using comfyui’s text encoder to encode prompt?No. comfyui uses its own text encoder. Automatic1111 uses Our backend is Automatic1111 and we use Q: Are you using comfyui’s UNet implementation?The UNet implementation (LDM) uses patches from 3 sources: Automatic1111, ComfyUI, and Forge. Note that Automatic1111 uses lots of monkeypatch to implement things, so that the patches are in runtime and you cannot find those codes rightly near the codes of UNet. ComfyUI and Forge directly add codes to Unet so you can find codes directly in the UNet file. We also need to point out that the top-level SD object in Forge/Automatic1111 are Also, note that Forge uses standard The LDM’s patches from Automatic1111, ComfyUI, and Forge are equally important in this project. Q: Can you give an overview of your real backend?Forge backend = A1111’s sampling + A1111’s text encoder (clip, emphasize, prompt scheduling, etc) + A1111’s Stable Diffusion Object And, we believe that the claim “Forge is using ComfyUI as a backend” is very misleading. Q: Why not import comfyui as a whole in this repo? Why use other file structures?To remove duplications. As in the question “Can you give an overview of your real backend”, you can see that most codes from comfyui are imported but not used. We re-organized the code structure to reduce duplication. However, even after reorganization, many unused codes still exist. Q: Why I can find codes same as comfyui in Forge?See also "Q: Can you give an overview of your real backend?", "Q: Why not import comfyui as a whole in this repo?", "Q: If you do not use them as backend, why you import them?" Q: Do you give credits to comfyui codes?Yes. Several days ago, the licenses are in the credits are at (Note that this is given to every file that has an entry to the imported codes, i.e., comfyui nodes.) And now credits are given to every code pieces. Q: Can I transfer a comfyui extension to Forge’s Unet patcher?Yes, but this needs some efforts to change the API format. Also, Forge’s UNet Patcher system natively support instant-id, reference only, animatediff’s block insert. These needs monkeypatch in comfyui (2024 Fed 11). And do give credits to comfyui codes with original sources and attach the license. |
Beta Was this translation helpful? Give feedback.
-
|
Oh man, whats the award for the winner ?I had work stolen so many times , its so petty to call everyone out now , so... in the end, is it really THAT important, is it top secret code ? The only thing people see is the attitude about it. |
Beta Was this translation helpful? Give feedback.
-
|
it's fine to use other's code. to publicly deny it when your repo says otherwise is just wrong. again, this is not about attribution but about what you tell people publicly. the part about comfy not being able to do BREAK and timesteps is also false so I'm not sure why @lllyasviel even mentioned that if he himself said he is not familiar with using comfy despite using it for fooocus. the way he conducted himself during this discussion very much seemed like he is trying to demerit the work of others rather than talking about how comfyui is used in the backend. I still think the title and opening of this discussion is misleading without reading the full context. this would still be better closed and posted fresh as not to confuse people reading it. I have already contacted github support for closing this discussion unless @lllyasviel does it himself as well as remove the damaging commit comment here ee023f4.
this isn't entirely true. the backend is mainly A1111 but with additions from Comfyui, diffusers, and the forge team. please follow through with what @KohakuBlueleaf and @continue-revolution have stated before in a new announcement and pin that. |
Beta Was this translation helpful? Give feedback.
-
|
Dude, you’re a stinker, it would be nice if you stop acting high and might. Both you and comfy annonomus sound whiny. You’re also being passive aggressive in your language and it is off putting. People within the development community see these things. Your reputation is being blasted because of your snarky responses. Sound familiar, anyone else see what I did there? |
Beta Was this translation helpful? Give feedback.
-
|
All I can say is don't let this discourage you and don't let it slow you down, because this is a great project. I think it's more than acceptable for the A1111 team to handle the core of the web UI, and the Forge team to handle its optimizations, and also it's more than understandable to be done as a fork. Just like the namesake for the fork, with Mojang and the community, it's very common to do it this way with software. |
Beta Was this translation helpful? Give feedback.
-
|
Why are my idols fighting each other 😭 |
Beta Was this translation helpful? Give feedback.
-
|
people responding to people saying that those people steal code. did they steal the code? i'd rather not start an argument so you decide |
Beta Was this translation helpful? Give feedback.
-
|
People aren't even debating about if it's stealing or not, they're debating if ComfyUI is used as the backend. Very sad to see. |
Beta Was this translation helpful? Give feedback.
-
|
It's all pedantry, boasting, arguing semantics and double-speak instead of up-front clarity, recognition of missteps and comradery. Yes, very sad to see in any project, and worse, sad to have my attention drawn to it when I'm simply trying read the readme before using it. |
Beta Was this translation helpful? Give feedback.
-
|
From an outsider perspective, also a non-coder, this discussion looks like a drama caused by different definition and understanding of "word". |
Beta Was this translation helpful? Give feedback.
-
|
Um... not to add fuel to a fire under a big nothing burger... but... could I encourage you @lllyasviel to create an option to use a folder structure like Comfy. It's a real nightmare trying to share all the data between Automatic1111 or SD-Forge and ComfyUI. SD-Forge is still fairly young and a switch that would redirect all scripts to the appropriate Comfy folders would allow you to share a model folder setup for both. Might help in the adoption of SD-Forge as a quick way to iterate/explore vs Comfy which is more... workflow oriented. |
Beta Was this translation helpful? Give feedback.
-
|
Comfy can handle the a1111 folder layout if you add it to a config, so you can keep stuff in a1111 format folder while using them in Comfy still. It's not the same thing but it's something if you want to use both at once |
Beta Was this translation helpful? Give feedback.
-
|
True… but I like the organization of Comfy lol… even though I started with Automatic1111 |
Beta Was this translation helpful? Give feedback.
-
I use comfy, fooocus, forge, auto1111 and share all the resources with each other. I'm not really sure of this "organization" you speak of that comfy has that the others don't? Are you talking specifically about the UI, or nodes? Because Forge/Auto1111/comfy have a drop down menu listing checkpoints, a folder structure option I use for checkpoints/LoRas/embeddings, and the kicker for me personally as to why I struggle using comfy for anything I am not intimately familiar with - display pictures for everything with an extra extension. It's been a long time since I've used comfy as I much prefer forge for all the reasons I listed above - but I do go back and play with it from time to time to see what changes happen in the gens by using a different backend. Edit Reading again it looks like just sharing recourses? |
Beta Was this translation helpful? Give feedback.
-
|
@lllyasviel keep up the good work. All of your work is top notch and I can’t wait to see where you take forge. Good job giving credit, and good job professionally explaining your point. |
Beta Was this translation helpful? Give feedback.
-
|
I support you, author. Your forge is very useful. You know that the domestic "Qiuye integration package developer" is slandering you. This is a manifestation of incompetence. |
Beta Was this translation helpful? Give feedback.
-
|
Folks, I read this and feel back in the days when I started doing a UI for SD... I messed it up and it never fly. But I got fired at by many people too for "using their code" which sometimes it was a single line in a vast library they used from somewhere else... Sometimes I don't even know those lines where in there.... I get that thing about tell what you use and please add my name to your product. But really? its open source and you give it out for free and there seems to be any credit inside the codes that was asked for, the time you spend blaming each other should be used drinking beer together or eat pizza or whatever you like. or even better spend on developing new ideas. In the name of science here, stop throwing shit at each other and do the things you are good at, writing brilliant unprecedented code the world will use for fun. If it helps I invite both of you to pizza and beer in my garden until it got solved... |
Beta Was this translation helpful? Give feedback.
-
|
I also read this discussion and don't really understand the point in this argument. If both solutions are open sourced, they are exactly that: open sourced. To me it means that you are free to read the code and use the code, unless the license states otherwise, limits that ability, enforces performing certain necessary steps beforehand for the proper use of said license, so on and so forth. If some points of the other open sourced software's license(s) were not realized properly, the given open source author utilizing that other open sourced software's code should be informed about such fact and worked with on a solution to the problem. I don't see Github as the right place to be arguing about the things mentioned above, even repeatedly so. This may be off putting to young coders/researches that may want to fork a project to work their skills at. So please, avoid unnecessary arguments that make it seem like we're crossing a barrier of an open sourced world of this place. |
Beta Was this translation helpful? Give feedback.
and others
-
Recently some people begin to spread misinformation about Forge using ComfyUI as a backend.
This is false, harmful to the community, and harmful to the efforts of our engineering team.
The backend is WebUI backend with Forge optimizations. (We call it Forge backend.)
If you see similar misinformation again, please stop spreading it, and if necessary, post a link to here.
Use this test as an example:
Prompt
fantasy landscape with a [mountain:lake:0.25] and [an oak:a christmas tree:0.75][ in foreground::0.6][ in background:0.25] [shoddy:masterful:0.5]
BREAK
a beautiful landscape, (nice:0.9) and (cool:0.5), (trees:0.4), (flowers:0.3), (lake:0.5)
BREAK
masterpiece
Negative Prompt
bad, (ugly:2.0), low quality, (bad anatomy:0.7), (signature:0.8), watermark, (username:1.1), (error:1.2), (missing limbs:1.3), error
This is WebUI:
fantasy landscape with a [mountain:lake:0.25] and [an oak:a christmas tree:0.75][ in foreground::0.6][ in background:0.25] [shoddy:masterful:0.5]
BREAK
a beautiful landscape, (nice:0.9) and (cool:0.5), (trees:0.4), (flowers:0.3), (lake:0.5)
BREAK
masterpiece
Negative prompt: bad, (ugly:2.0), low quality, (bad anatomy:0.7), (signature:0.8), watermark, (username:1.1), (error:1.2), (missing limbs:1.3), error
Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 12345, Size: 512x512, Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0-460-g658da35c
This is Forge:
fantasy landscape with a [mountain:lake:0.25] and [an oak:a christmas tree:0.75][ in foreground::0.6][ in background:0.25] [shoddy:masterful:0.5]
BREAK
a beautiful landscape, (nice:0.9) and (cool:0.5), (trees:0.4), (flowers:0.3), (lake:0.5)
BREAK
masterpiece
Negative prompt: bad, (ugly:2.0), low quality, (bad anatomy:0.7), (signature:0.8), watermark, (username:1.1), (error:1.2), (missing limbs:1.3), error
Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 12345, Size: 512x512, Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: f0.0.11-latest-107-g44b647a8
You can see that they are same images (this webui is on xformers, and forge is on sdp).
This example uses almost all "Automatic1111-only" features like A1111's unique BREAK, A1111's unique prompt emphasize, A1111's unique prompt timestep sceduling, etc. Also read:
https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#prompt-editing
https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#break-keyword
https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#attentionemphasis
TL;DR: [mountain:lake:0.25] means the first 25% steps use the word
mountainand latter 75% steps use the wordlake.To the best of my knowledge, although ComfyUI has lots of nodes to change GPU generator or emphasize method that may be able to approximate the similar things, getting this same results with all "Automatic1111-only" features is still impossible. ComfyUI does not have natively enabled prompt timestep scheduling, BREAK, etc. And different emphasizing method or different text encoder implementations will lead to different results.
We imported many codes from diffusers but do not use most of them. This is same for comfyui. You can find codes similar to comfyui but (1) most are imported just like importing another diffusers but they are not used and (2) we only use comfyui's implementation reference to build the memory management and UNet Patcher system (so that the the API like
unet.add_xxx_patchwill be very similar and extensions can transfer to each other, to reduce the learning cost of developers).Besides, all codes are open-sourced and every one can review the implementation.
We reserve the right to defend our work against any misinformation in a more formal way if necessary.
lllyasviel
Beta Was this translation helpful? Give feedback.
All reactions