Skip to content

ControlNet

Jump to bottom
Ben Rugg edited this page Apr 14, 2023 · 5 revisions

Introduction to ControlNet

ControlNet is an extension for Automatic1111 that provides a spectacular ability to match scene details - layout, objects, poses - while recreating the scene in Stable Diffusion.

At the time of writing (March 2023), it is the best way to create stable animations with Stable Diffusion.

AI Render integrates Blender with ControlNet (through Automatic1111's API), allowing an easy and quick way to transform your scenes, sketches or videos.

AI Render ControlNet

Make sure to update AI Render or get the latest release.

ControlNet Installation

Prerequisite: You must have Automatic1111 already installed.

  1. In the Automatic1111 web ui, go to the Extensions tab, then Install from URL, then paste https://github.com/Mikubill/sd-webui-controlnet into the url for git repository, then click Install.

  2. Under Extensions > Installed, click Apply and restart UI

  3. Download one or more model files from https://huggingface.co/lllyasviel/ControlNet/tree/main/models. (You do not need all of them. You may want to start with just the depth model. See below for more info about the different models).

  4. Move the downloaded model file(s) to your Automatic1111 directory on your computer, in this subdirectory: extensions/sd-webui-controlnet/models

If everything has been installed correctly, you can now go to Blender, find the AI Render panel, and then click Load Models from Automatic1111 under the ControlNet panel. If you see two dropdowns with selections for preprocessor and model, then you're good to go.

Video tutorial for installation (not affiliated with AI Render): https://www.youtube.com/watch?v=mhJa1dhIFWg

ControlNet Models

...coming soon: better info here.

For now, you can see the different models here: https://github.com/lllyasviel/ControlNet

TL;DR: For Blender, using depth (for both the preprocessor and model) will probably provide good results for many scenes. Use openpose for characters/people. mlsd should be good for architecture. Try experimenting with normal, canny and sketch, as well. Please let me know what gives you the best results!

Troubleshooting

...coming soon.

For now, make sure you have ControlNet successfully installed. You should be able to use it through the Automatic1111 UI.

Please submit a bug report if you run into anything.

Clone this wiki locally