Ideas and more... (Long)

[gateway]

So I finally got a chance to mess around with generating some scripts that I want to try. Now I have to say im not a python programmer more of hacker with python, my coding really is web based style stuff but my background is a video game developer starting back in the early 1990's. I have been though various industries and startups but am always someone that loves to take technology and create art with it when possible. Sorry had to get it out of way because what I may share with code you may vomit :)

Here are the ideas I'm trying to create and evolve and some already added to my hack and slash code

Singe Images:

RAW thoughts.

• Pass in a text file of text prompts and loop over them (pretty easy, just havent done it)
• Pass in or load a text that define art styles (8k, pencil sketch) added as an array to my code to enable for a single image and iterate over all the styles pass on the command line -a True
• Added command line args (easier for me to handle what im wanting to do, when tweaking settings constantly)
• Command line arg for folder that also upscaled images use
• Command line arg to enable upscale
• Ability to properly size the min and max image size based upon init image size (landscape, portrait) my max width or height depending on image is 900... so 900 x H, W x 900 I hacked a small function for this but not in single image.
• Export and create photo collage with some meta data
• Export out html file read more below on html

I like to take a single with text or with test + image ref and have it iterate over text prompts, styles etc..

A example would be to take the Nike shoe I did and mixed with a unicorn to break out text prompts into a few things,.

Text Beginning + Material Type + Style Type iterate over all of those till done.

• "A Nike shoe made out of + Material"
• "A Nike Shoe made out of + Material + Style"

This gives me a folder of the various tries and again allowing me to pick what I like the best or what looks the coolest.

ideally after that is finished generate a montage but montages have limits so it got me thinking why not output each folder that created and generate a html file

HTML

• If single image generate the filename based upon the text prompt that stays the same but has different materials or styles
• title tag as a text prompt
• Create a responsive image gallery with captions that have the extra meta data
• Clicking on an image if upscaled was enabled would show the large image (maybe need to resize down a bit) and show the image in a slider.
• Add new page created to a master index file

If you have different ext prompts the page layout could be a row and each column has all the meta data on the left and image on the right..

I'm still working this out in my head and I'm not sure if you care but just thoughts that I want to do while I cant do all the cool coding using the engine functions like you so far :)

So here is an example of adding an array of styles and just looping though them.. this is "A painting of Han Solo + Style"

umm code..

import vqgan_clip.generate
from vqgan_clip import esrgan
from vqgan_clip.engine import VQGAN_CLIP_Config
import os, argparse

# Create the command line parser arguments.
parser = argparse.ArgumentParser(description='Single Image Styling')

parser.add_argument("-p",   "--prompts", type=str, help="Text to pass into the system", default="A painting of a mushroom on psychedelics", dest='prompts')
parser.add_argument("-ip",  "--image_prompts", type=str, help="Images that will be turned into a prompt via CLIP", default=[], dest="image_prompts")
parser.add_argument("-np",  "--noise_prompts", type=str, help="Random number seeds can be used as prompts using the same format as a text prompt. E.g. '123:0.1|234:0.2|345:0.|3' Stories (^) are supported.", default=[], dest="noise_prompts")
parser.add_argument("-ii",  "--init_image", type=str, help="Seed Image that can be used to start training with", default=None, dest="init_image")
parser.add_argument("-i",   "--iterations", type=int, help="Number of iterations of train() to perform before stopping and outputing the image. 200-300 works fine", default=100, dest="iterations")
parser.add_argument("-se",  "--save_every", type=int, help="An interim image will be saved to the output location every save_every iterations", default=50, dest="save_every")
parser.add_argument("-cp",  "--change_prompt", type=int, help="Serial prompts, sepated by ^, will be cycled through every change_prompt_every iterations.", default=0, dest="change_prompt_every")
parser.add_argument("-o",   "--output", type=str, help="Location to save the output image file when a single file is being created.", default='output.png', dest="output_filename")
parser.add_argument("-s",   "--image_size", nargs=2, type=int, help="x/y dimensions of the output image in pixels. This will be adjusted slightly based on the GAN model used. VRAM requirements increase steeply with image size.", default=[256,256], dest="output_image_size")
parser.add_argument("-in",  "--init_noise", type=str, help="Seed an image with noise. Options None, pixels or gradient", choices=['pixels','gradient'], default=None, dest="init_noise")
parser.add_argument("-iw",  "--init_weight", type=float, help="A weight can be given to the initial image used so that the result will 'hold on to' the look of the starting point.", default=1.0, dest="init_weight")
parser.add_argument("-opt", "--optimizer", type=str, help="Optimizer Options", choices=['Adam','AdamW','Adagrad','Adamax','DiffGrad','AdamP','RAdam','RMSprop'], default='Adam', dest='optimizer')
parser.add_argument("-d",   "--decay", type=str, help="Method used to compare current image to init_image.", choices=['decay','original'], default='decay', dest='init_weight_method')
parser.add_argument("-sd",  "--seed", type=int, help="Seed Value to pass in elese its random", default=None, dest='seed')
parser.add_argument("-us",  "--upscale", type=str, help="Enable Image upscale", choices=['True','False'], default='True', dest='upscale_image')
parser.add_argument("-f"    "--folder", type=str, help="Output Folder Location default output", default='output', dest='folder')
parser.add_argument("-a"    "--art_types", type=str, help="iterate though the art types for your image", default=False, dest='art_types')

#execute the arts statements.
args = parser.parse_args()

#create something
if not args.prompts and not args.image_prompts:
    args.prompts = "A question mark made out of a mushroom on psychedelics"

config = VQGAN_CLIP_Config()
#config options from arguments else default.
config.seed = args.seed
config.output_image_size = args.output_image_size
config.init_weight = args.init_weight
config.init_image = args.init_image
config.init_weight_method = args.init_weight_method
config.init_noise = args.init_noise
config.optimizer = args.optimizer
text_prompts = args.prompts

# Enable image to be upscaled
upscale_image = args.upscale_image

art_types=["8k resolution","pencil sketch","8K 3D","deviantart","CryEngine","Unreal Engine","photo illustration","pixiv","Flickr","Artstation HD","Behance HD","HDR","anime","Ambient occlusion","Global illumination","Chat art","Low poly","Booru","Polycount","Acrylic Art","Hyperrealism","Zbrush Central","Rendered in Cinema4D","Rendered in Maya","Tilt Shift","Mixed Media","Detailed painting","Volumetric lighting","Storybook Illustration","#vfxfriday","Ultrafine detail","matte painting","Watercolor","CGSocity","child's draing","marble sculpture","airbrush art","renaissance painting","Velvia","dye-transfer","stipple","Parrallax","Bryce 3D","Terragen","Bokeh","charcoal drawing","commission for","polished","aftereffects","datamosh","holographic","dutch golden age","digitally enhanced","Art on Instagram","bokeh","psychedelic","wavy","groovy","movie poster","pop art","made of beads and yarn","made of feathers","made of crystals","made of liquid metal","made of glass","made of cardboard","made of vines","made of flowers","made of insects","made of mist","made of paperclips","made of rubber","made of wire","made of trash","made of wrought iron","tattoo","woodcut","American propaganda","Soviet propaganda","Fine Art","Photorealism","drone shot","poster art","Impressionism","Lowbrow","Egyptian art"]

if art_types:
    print('in art types ========================')

    for x in range(len(art_types)):

        art_prompts = text_prompts + " " + art_types[x]
        vqgan_clip.generate.single_image(eng_config = config,
            text_prompts = art_prompts,
            init_image = args.init_image,
            iterations = args.iterations,
            save_every = args.save_every,
            output_filename = os.path.join(args.folder, args.output_filename + art_types[x]))

        #Upscale image
        if upscale_image:
                esrgan.inference_realesrgan(input=os.path.join(args.folder, args.output_filename + art_types[x] +'.png'),
                    output_images_path='output_upscaled',
                    face_enhance=False,
                    purge_existing_files=False,
                    model_filename='RealESRGAN_x4plus_anime_6B.pth',
                    model_url='https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth')
else: 
    print('not in art types ========================')
    vqgan_clip.generate.single_image(eng_config = config,
        text_prompts = text_prompts,
        init_image = args.init_image,
        iterations = args.iterations,
        save_every = args.save_every,
        output_filename = os.path.join(args.folder, args.output_filename))

    #Upscale image
    if upscale_image:
            esrgan.inference_realesrgan(input=os.path.join(args.folder, args.output_filename + '.png'),
                output_images_path='output_upscaled',
                face_enhance=False,
                purge_existing_files=False,
                model_filename='RealESRGAN_x4plus_anime_6B.pth',
                model_url='https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth')

Final Thoughts

Anyhow for single images for now my idea is to process the following

• Text Prompt Fragment that doesn't change and looped though + Materials and or Styles
• Various Text prompts looped though + Materials and or Styles
• Ability to cycle though iterations, init_weight etc.
• A good visual way to look at all output results first step would be montage (but would be huge)
• A static HTML page that can add a lot of meta data to (much more my idea - Looking at python template engine.

Video I have tried a few things but something is just wrong or my results suck (to much fast changing artifacts between frames)

Issue

One issue I have seen so far.. ESRGAN doesn't keep the meta data so need to figure out if its possible or to re-add it upon save.

Max Resolution

Properly size to max gpu based upon init image

Function to properly match init image if used based upon your max gpu output (I know ESRGAN upscales but when you have a higher res image to start with its better for me at least)

I have this as a test the init image is then passed though this and my W x H is generated (Im not sure if your doing this)

def get_size(filename: str) -> str:
    #load image
    im = Image.open(filename)
    w, h = im.size
    #define what size I want to output based upon Titan RTX
    if w > h:
        #landscape
        width = 900 #max for my gpu
        height = round(width * h / w)
    elif w < h:
        #portrait
        height = 900
        width = round(height * w / h)
    else:
        #square
        width = 900
        height = 900

    return width, height

[rkhamilton]

I'm glad to see you're fired up to use the tool!

A few thoughts as I read:

• If you're building a wrapper to enable running from the shell, check out the NerdyRodent repo that I was initially inspired by. He has a shell script for generating combinations of words to create random prompts.
• I don't know anything about html coding, so I'd be interested in any tools you create for creating galleries that incorporate metadata.
• Your algorithm for procedurally generating styles looks reasonable to me! I can suggest a couple of python techniques to take advantage of if you're interested:
  • Python offers a modern way to iterate over iterables without counting indexes.

    for x in range(len(art_types)):
        print(art_types[x])
    # is the same as, the pythonic way
    for art_type in art_types:
        print(art_type)
    # the second version is more clear, and also prevents off-by-one errors when looping over iterables.

  • You can wrap lines on lists if you start a newline at a comma.

    art_types=["8k resolution","pencil sketch","8K 3D","deviantart","CryEngine",
            "Unreal Engine","photo illustration","pixiv","Flickr","Artstation HD",
            "Behance HD","HDR","anime","Ambient occlusion","Global illumination",
            "Chat art","Low poly","Booru","Polycount","Acrylic Art","Hyperrealism",
            "Zbrush Central","Rendered in Cinema4D","Rendered in Maya","Tilt Shift",
            "Mixed Media","Detailed painting","Volumetric lighting",
            "Storybook Illustration","#vfxfriday","Ultrafine detail","matte painting",
            "Watercolor","CGSocity","child's draing","marble sculpture","airbrush art",
            "renaissance painting","Velvia","dye-transfer","stipple","Parrallax",
            "Bryce 3D","Terragen","Bokeh","charcoal drawing","commission for",
            "polished","aftereffects","datamosh","holographic","dutch golden age",
            "digitally enhanced","Art on Instagram","bokeh","psychedelic","wavy",
            "groovy","movie poster","pop art","made of beads and yarn","made of feathers",
            "made of crystals","made of liquid metal","made of glass","made of cardboard",
            "made of vines","made of flowers","made of insects","made of mist",
            "made of paperclips","made of rubber","made of wire","made of trash",
            "made of wrought iron","tattoo","woodcut","American propaganda",
            "Soviet propaganda","Fine Art","Photorealism","drone shot",
            "poster art","Impressionism","Lowbrow","Egyptian art"]

• I posted another reply to you that had some scripts attached that you could see as examples of building iterables for a bunch of parameters at once.
• As far as fast changing artifacts between frames, check out the z smoothing in v1.3.0. It helps a lot, at the expense of some blur on rapid movement. I also had no problem getting RIFE installed in the save venv as vqgan, and it runs fine to apply optical flow smoothing on videos I generate. If you use RIFE (or other OF) you would probably want to generate a lower framerate video and then let OF fill in the gaps in the frames. E.g. generate 15fps, and let OF convert to 60 FPS. Also be aware that RIFE seems to save uncompressed mp4s. I didn't look closely but it didn't seem to be h264. Expect to recompress with ffmpeg.
• If you come up with a script to copy the PNG metadata over to the Real-ESRGAN upscaled images, send it along and I'll incorporate it.
• I have a function built in for calculating the output image size based on an aspect ratio of an existing file (filesize_matching_aspect_ratio) that I use for restyling video. I didn't think to use it in single_image, but that makes sense. I just made the change. I'll push it to v.1.3.1 by the time you read this. The code you have at the bottom is creating unnecessarily small files for you. If you can do 900x900, then you can do wider than 900 in 16:9 aspect ratio. Feel free to copy this function from _functional.py. To use it like in your example code:

  from vqgan_clip import _functional as VF
  width, height = VF.filesize_matching_aspect_ratio(template_image_path, 900, 900)

Keep me posted how it's going!

[gateway]

Thanks for taking the time to reply..

I tend to spit out a lot of thoughts all at once and some times not fully formed yet.

I do want to work on something that reads a folder dir and creates static html pages that can be viewed locally or uploaded to some web server if you want. I may use a template engine so I dont hav to write stuff from scratch.. probably go with this https://jinja.palletsprojects.com/en/3.0.x/templates/ Ill start to poke around with it this week but I'm trying to wrap up a bunch of stuff so it will be slow ..

Sizing:
Thanks for putting in the resizing function, I need to play around with what max I can do at 16:9 and just throw incremental sizes at he gpu till it runs out of memory.

Thanks for suggestions on ways to iterate over loops etc.

Video

I have tested rife and will run some more tests using lower frame rates and having it interpolate the in between frames. and hopefully that works out but this will totally remove any ability to do proper audio with video (I want to do a short video of one of my friends bands).

[rkhamilton]

You would still be able to use rife + audio. Run the VQGAN extract at lower framerate, say 15, then use my package encode too with the same input framerate of 15. The resulting video is the same duration as the original, but at only 15 fps (so you could copy audio no problem). Then run that generated video through rife, and it will pop out at 60fps, but the same duration. I can't guarantee the duration will be exactly the same, but I expect it to be really close.

re: sizing, the intent of my function is that if you can run a 900x900 image, just run that example code with a 16:9 image and it will tell you your max image size at 16:9. You might be able to run a touch higher, but you might also be able to run a touch higher than 900x900 in square format.

Speaking of ideas: I think I've got a breakthrough on smoothing generated frames in a novel way. I'm running a bunch of tests and should push a new version in a day or two, but it's really promising.

I'm probably going to bundle that with some changes to style transfer that will be a new syntax that should make content creation easier. I'm thinking that instead of specifying iterations and save_every, you would specify how many frames you want to generate, and how many iterations per frame. I had just copied other notebooks, but really, nobody cares how many total iterations are run, you want to generate some video footage. So if you want 10 seconds at 15p, that's 150 frames. So, something like below. It should make calculating durations easy.

generate.video_frames(config, num_video_frames=150, iterations_per_frame=30)

I'm also going to get rid of multiple_images (you can just as easily loop over single images) and get rid of the non-zoom video function. The video function is complex and I don't want to keep them both updated. If you run zoom_video with zoom_ratio=1.0 it works just like the other one.

[gateway]

Cool I like the way you think.. I was up all night thinking about other ways to make it more flexible for artistic creations, I kinda link it to some sort of old synth back in the days where you would tweak, and turn knobs to get the best result but at least you had enough buttons or knobs to play around with and iterate till you get your sound.

So I think we have a few things, iterations, weight, noise, init image, various text prompts, zoom features, blend features, etc.. ideally these days you could say just create a 100 images for me with various settings w/o me thinking about it so I can then look at them and choose the best one. Of course we would all like x and y to happen with everyday things in our lives.

So it would be cool to feed in a text file of prompts (I could prob hack that in for myself) but it when it comes to giving me a lot of images at different values thats where I sorta fall short.

Upscale Image meta

If you have a wrapper for it , I think that each time a frame is upscaled we know the source folder with the meta data, so if we store that, let the upscaler do its work and use PIL to load and resave the image with the metadata it would all be automatic.

1. upscaler does image 1.png, put it in upscaler folder, another block of code runs
2. load original png get meta data out
3. load upscaler image and and resave it

?

[rkhamilton]

I'll include a couple of example scripts in v2.0 that will generate multiple images that vary some parameters. There's not a lot of degrees of freedom in creating single images. I'll provide a script that creates multiple shots on goal using different seeds, and a script that creates images using your list of prompt keyword phrases. The really complex one is style transfers on source video. I'll try to make a tool to explore options for that as well.

I'll look into something for PNG metadata.