Controlnet Sketch and Scribble not working as intended?

[FreeeFry]

I'll start this as a discussion instead of a bug report since I don't know if this is the intended behavior or not;

It seems that in forge with the integrated Controlnet both the Sketch and Scribble preprocessors aren't working the same way they do in regular a1111 webui in that they only seem to accept uploaded images and not support drawing manually on the canvas neither by creating a blank canvas and drawing on it or uploading an image and drawing on top of the image.

Is anyone else experiencing this?

For example
Using cyberrealistic_v41BackToBasics checkpoint with prompt a large grassy field, a house in the center of a grassy field, realistic Neg: art, 3d, cartoonand seed 4272455667 with the following Controlnet scribble:

The image produced is different than without controlnet enabled but doesn't at all resemble the shape scribbled:

With Controlnet:

Without:

The scribbled image is shown as a separate image though in the output so it's at least being intercepted from the Controlnet input it seems;

[FreeeFry]

If I save the Scribbled output image and load the image into Controlnet it works, so it seems that drawing on the canvas itself specifically is what is not working properly?

This is how the generated images looks with the scribble being uploaded as an image instead:

[cscooper2000]

I'm having the same issue. In a1111 my workflow consisted of rendering a decent image and then loading it into the ControlNet (scribble) as a "reference image" (not sure of the terminology). Now I could scratch in a few specific details in any area of the photo and like magic, I could create the exact details I intended.

But, yeah... In Forge it doesn't seem to work at all. I can still load an image, but tracing or sketching in new details has zero effect. :(

[ostap667inbox]

That's your mistake, not ControlNet's :)
By drawing a sketch sketch on a black background you are essentially doing the work of a preprocessor. That is, you MUST select 'none' in the 'preprocessor' field. In this case, the model will work directly with your sketch. Instead, you are trying to get the preprocessor to make its own sketch from your sketch. Of course, nothing productive will come of it.
A preprocessor is needed when you use a photo, or other full-fledged image, as your source image.

[FreeeFry]

No, it is how it used to work (and afaik. still works) in the original implementation for automatic1111's webui, the new behavior is different and arguably a lot worse as it means you have to do any sketching using a reference image outside in an external editor and save that to an image which imho. defeats most of its use. @ostap667inbox

I really hope this is not actually intended behavior..

[ostap667inbox]

The canvas in built-in ControlNet has a bug, you're right. It is not passed to the preview. But instead you can draw directly in the browser in the preview window. In Forge, the Photopea script is built into ControlNet:

This solution definitely works:

It's strange that your screen doesn't show this button. Maybe photopea is disabled in your settings?

[FreeeFry]

I had disabled Photopea as I didn't like having my local webui connect to an external service.

I re-enabled it now to test this, but it doesn't work for me in Firefox at least, none of the edits I make are being sent back into the preprocessor preview. I feel it's a bit beside the point though as the original workflow that worked perfectly in a1111 was to simply upload a reference image into the preprocessor, draw on top of it and use the underlying reference as guide for details, etc.

I should perhaps have used an actual example in my initial post instead of drawing on top of a blank canvas, but imagine instead those drawings being on top of a reference image such as that of a grassy field or some such.

[FreeeFry]

I booted up and updated a1111 to latest version and retried this again and yeah it works pretty much as expected.

Poor drawing skills aside, this is a pretty nice workflow imho.
Sketch on top of a pre-rendered reference:

Scribble pidinet processed:

Cherry picked image out of 4:

[makehillflat]

Before responding that someone is incorrect, it might be beneficial to review the question more thoroughly. Are you referring to Forge, or Auto1111? According to the original poster (OP), the 'sketch (in place)' function seems to be missing from the Forge controlnet implementation, and my observations confirm this. The option to select 'none' in the 'preprocessor' field does not appear to resolve this issue, nor does any setting that allows for sketch input. It would be more constructive to focus on understanding and addressing the actual issue at hand rather than seeking to correct others. If the solution is unclear, perhaps further investigation or assistance from others could be useful.

[ostap667inbox]

I suggested another solution above. Sorry, misled and triggered by this:

[cscooper2000]

Just a bit off-topic... You mentioned Photopea. I was unaware of that until now. You say it uses an internet connection to send data to their service? But I'm reading otherwise online. Any clarification around this?

Either way, it looks pretty cool.

[FreeeFry]

It tells you when you try to use it;

you are about to connect to https://photopea.com