Parse LoRA from prompt

[Danamir]

This adds the capacity to parse LoRA directly from the prompt, using the Auto1111 webui syntax : <lora:name:strength>.

The lora name is searched in all the available lora models in the ComfyUI installation, by case insensitive filename only. No path, no extension, to keep the prompt light.

The strength is optional, replaced by 1.0 if absent.

Once applied the whole <lora*> is removed from the prompt before applying the conditioning.

For example the prompt photography of a bird <lora:add-detail-xl:1.0> auto loads the matching LoRA LoRA\5. SDXL\Style\Add-detail-XL.safetensors found on my installation :

If the LoRA is not found in the available models, an error is returned :

Same concept if an invalid strength is given :

I find it more responsive to display the error directly in the krita interface, but it could be replaced by a log line only in client.log

[Danamir]

The lint check outputs :

But I'm not sure what lines needs to be reformatted. I tried to apply an auto format in IntelliJ IDEA, but many lines are impacted, some not concerned by this PR.

Any idea ?

[edit] : I read the lint output too quickly, it suggests the modifications to be done.

[Acly]

This looks nice to have functionality wise, I like the error handling too.

Question: since comfy uses embedding:file syntax for embeddings, does it make sense to support lora:file? It's a bit leaner, but doesn't work for filenames that contain spaces I guess.

Requests:

• Since prompt weight/strength can now be changed with Ctrl+Up/Down, it would be nice to have this for LoRA. From a user perspective I would kind of expect it to work. Maybe it's enough to add support for <> back in, or the extra colon needs to be handled.
• Please write a test for parse_lora (you may want to take a lora list as argument rather than Client)

[Danamir]

Question: since comfy uses embedding:file syntax for embeddings, does it make sense to support lora:file? It's a bit leaner, but doesn't work for filenames that contain spaces I guess.

There is the problem with LoRA names allowing spaces, and the attention edit on Ctrl+up/down which relies entirely on delimiters. (cf. below).

Since prompt weight/strength can now be changed with Ctrl+Up/Down, it would be nice to have this for LoRA. From a user perspective I would kind of expect it to work. Maybe it's enough to add support for <> back in, or the extra colon needs to be handled.

Done. I set the lower bound to -2.0 to allow negative LoRA strengths, and patched the parser to allow strength 1.0 to be displayed for <> delimiters.

Please write a test for parse_lora (you may want to take a lora list as argument rather than Client)

Done.

[Acly]

Thanks! works very smoothly now

[peterpernhofer]

Sorry to say. But in Krita-Plugin version 1.14.1 it seems. that this (very very useful) feature is missing again.
Loras are not parsed from prompt. I have to add them manually in the connection dialog settings every time, I want to use a lora

[Danamir]

It's still working for me : tested with a prompt "a bird on a branch lora:add-detail-xl:1.0" and the option to dump the workflow I checked that the LoRA is correctly affecting the output :

Can you see if the dumped workflow is working (it can be found in %APPDATA%\krita\ai_diffusion\logs) ? Also check the logs to see if there is a name resolution error ?