REPL command returned after creating variation does not create the same image

[psychedelicious]

Describe your environment

• GPU: mps
• VRAM: 32GB Unified Memory
• CPU arch: arm64 M1
• OS: macOS
• Python: mamba env create -f environment-mac.yaml
• Branch: development
• Commit: i do not have a line that starts with "Merge", but I am up to date with development

Describe the bug
After generating an image, the REPL returns the command (parameters) which may be used to recreate that same image.

When variation amount is specified, the command returned does not recreate the same image.

To Reproduce

1. Run python scripts/dream.py
2. Generate an image using -v e.g. test -s 5 -S 0 -v 0.1, call it "Image 1"
3. It returns a command e.g. outputs/img-samples/000070.810716037.png: "test -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S 810716037 -n 1 -f 0.75 -V 810716037:0.1
4. Copy that command and paste back into the REPL
5. A different image is generated, call it "Image 2"

Expected behavior
"Image 1" and "Image 2" are identical.

Additional context
Maybe I am misunderstanding how variations work

[lstein]

This could either be a new bug I introduced during the RFC #266 work (i.e. the prompt string is wrong), or an older bug regarding reproducibility problems in random number generation on M1 systems. Could you compare the behavior against the main branch so that I can distinguish those possibilities?

[psychedelicious]

Same issue on main.

REPL command 1: test -s 5 -S 0 -v 0.1
REPL output 1: [1] outputs/img-samples/000079.1485615218.png: "test" -s5 -W512 -H512 -C7.5 -Ak_lms -V1485615218:0.1 -S0
Result image 1:

REPL command 2: "test" -s5 -W512 -H512 -C7.5 -Ak_lms -V1485615218:0.1 -S0 (same as output 1)
REPL output 2: [2] outputs/img-samples/000080.1485615218.png: "test" -s5 -W512 -H512 -C7.5 -Ak_lms -V1485615218:0.1 -S0 (same as command 2)
Result image 2:

Subsequent runs of the command yield identical images.

Does the expected behavior occur on non-M1 systems?

[lstein]

@mh-dm and @Any-Winter-4079 . Is this a manifestation of the non-reproducibility problem on MPS we were having earlier, or something separate?

I'm pretty sure I've tested reproducibility of variation on Linux and it is OK, but I'll check later today before declaring this to be an M1 specific bug.

[Any-Winter-4079]

@psychedelicious @lstein I did get the same image using
"test" -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S 810716037 -n 1 -f 0.75 -V 810716037:0.1
several times

---

Using this however
"test" -s5 -W512 -H512 -C7.5 -Ak_lms -V1485615218:0.1 -S0 I did not

---

It seems to me -S0 makes it not reproducible?
Another test
"test" -s5 -W512 -H512 -C7.5 -Ak_lms -V1485615218:0.1 -S3456

I'm using pytorch 1.12.1 by the way. Weeks ago, I had a reproducibility issue with the nightly, so I moved to 1.12.1. Not sure what version you are using.

Oh, another thing. I may not have the latest dev branch since I'm testing a bunch of things. But my version is at least consistent with the version from 1, max 2 days ago. Let me know if you are using 1.12.1 (so the problem is not nightly) and then I'll update and post my results again, to see if I can then reproduce the issue.

[lstein]

Ccing: @bakkot @blessedcoolant @xraxa

This turns out not to be an M1 specific problem, but a general issue for the variation-generating code on all platforms. If I run this identical command repeatedly I get different images even though I am setting the same seed each time:

dream> test image -S42 -v0.1

The culprit is L104 of ldm/dream/generator/base.pm, indicated by the arrow here:

    def generate_initial_noise(self, seed, width, height):
        initial_noise = None
        if self.variation_amount > 0 or len(self.with_variations) > 0:
            # use fixed initial noise plus random noise per iteration                                                                                                                                                                                       
            seed_everything(seed)
            initial_noise = self.get_noise(width,height)
            for v_seed, v_weight in self.with_variations:
                seed = v_seed
                seed_everything(seed)
                next_noise = self.get_noise(width,height)
                initial_noise = self.slerp(v_weight, initial_noise, next_noise)
            if self.variation_amount > 0:
==>          random.seed() # reset RNG to an actually random state, so we can get a random seed for variations                                                                                                                                           
                seed = random.randrange(0,np.iinfo(np.uint32).max)
            return (seed, initial_noise)
        else:
            return (seed, None)

random.seed() resets the random number generator using the system time. Therefore we will need to implement time travel in order to get back to the state as it was. This is seriously out of scope for this project.

Commenting out this line of code doesn't interfere with variation generation and solves one problem. Now I do get the same series of variations when using -v0.1 along with a fixed seed. Unfortunately a fixed seed and the indicated -Vxxxxxx value still stubbornly produces a different image from the original variant. So there is more work needed to explore this. I will check the main branch to see if anything is different there and will update this comment if so.

UPDATE: main does work as expected when I comment out the random seed generator, so it looks like I'll be able to track down the problem and fix it.

[Any-Winter-4079]

@lstein @psychedelicious Okay, just to comment what goes on in my machine:

"test" -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S 810716037 -n 1 -f 0.75 -V 810716037:0.1 which is what was originally posted produces the same result over and over.

test image -S42 -v0.1 does produce different results. But if you want to get one of the old results, you just need to use the same (second) seed -the ones coming from seed = random.randrange(0,np.iinfo(np.uint32).max)-, e.g. record it

get it

and then reuse it

It will generate the same image. The seed could be printed on screen, passed as an argument, added to the image metadata... Many possibilities.

So for example, you generate 100 images that are variations. Each with a different (second) seed. Want to recreate some of those images? Just reuse the (second) seeds for those specific images. I'm not sure a code that is supposed to generate variations of images, generating every time a different image based on the original is really a bug.
Especially when each image can be recovered/regenerated as well. Or am I failing to see something else?

Update. Example: with a fixed (second) seed = 4277317920 I can do:
"banana slice" -s5 -W512 -H512 -C7.5 -Ak_lms -v0.1 -S3454 -n3 and get

Then I can exit dream> q, re-enter
"banana slice" -s5 -W512 -H512 -C7.5 -Ak_lms -v0.1 -S3454 -n3 and get

[bakkot]

There are two things happening here:

• re-running with -v will produce different images
• using -S0 will generate a random seed, rather than using 0

The first thing is intentional: if you ask for 5 variations, then another 5 variations, you presumably don't want the same things you got last time. (So @lstein your commenting-out of the random.seed() line here should be reverted.)

The second thing is a bug caused (I expect) by this line - in python, 0 is falsey, so this line will reset the seed. You want to use if seed is not None rather than if seed here.

[Any-Winter-4079]

@bakkot I agree with you.

I think -S should indicate the seed for the original image. For example re-using -S3454 I expect the same image always
"banana slice" -s5 -W512 -H512 -C7.5 -S3454

If now I enter "banana slice" -s5 -W512 -H512 -C7.5 -Ak_lms -v0.1 -S3454 -n3 I expect -S3454 to only indicate that I want to generate random variations based on the image above. So I'll be creating 3 variations based on that image.

And if I enter that command again, I expect 3 new images, because -S3454 only refers to the original banana image (not to the variations -why regenerate the same variations?)

---

If we somehow want the same variations (which is not the normal behavior, but e.g. say because we liked one a lot), we need to keep track of the second seed).

[mh-dm]

Personally I would prefer "banana slice" -s5 -W512 -H512 -C7.5 -Ak_lms -v0.1 -S3454 -n3 to give repeatable results so I could go back to said results later if desired, or share the prompt. If I wanted more variations I could pass -n6. The downside of this is potentially repeated computation. (Note: just a personal preference)

[bakkot]

@mh-dm You can go back to those results later even when variations are randomized: each variation prints the full command necessary to reproduce that variation.

[Any-Winter-4079]

Personally I would prefer "banana slice" -s5 -W512 -H512 -C7.5 -Ak_lms -v0.1 -S3454 -n3 to give repeatable results so I could go back to said results later if desired, or share the prompt. If I wanted more variations I could pass -n6. The downside of this is potentially repeated computation. (Note: just a personal preference)

But that takes a lot of time. For example if I generate 20 images, then to generate a new one I have to specify -n21 because the first 20 images will be the same again...

[Any-Winter-4079]

@mh-dm You can go back to those results later even when variations are randomized: each variation prints the full command necessary to reproduce that variation.

That seems to not work yet because there is a seed totally random. seed = random.randrange(0,np.iinfo(np.uint32).max) but if we allow to either use this random seed or use an old one, then we can reproduce old variations.

The simplest way to test it is printing that second seed.

seed = random.randrange(0,np.iinfo(np.uint32).max)
print('seed', seed)

And then simply use it.

# seed = random.randrange(0,np.iinfo(np.uint32).max)
seed = XXXXXXX

What should happen is to add a new param here def generate_initial_noise(self, seed, width, height, seed_var=None): to specify that second seed, let's call it seed_var.
And then

if seed_var:
   seed = seed_var
   # we regenerate same variations
else:
   seed = random.randrange(0,np.iinfo(np.uint32).max)
   # we create random variations

[mh-dm]

But that takes a lot of time. For example if I generate 20 images, then to generate a new one I have to specify -n21 because the first 20 images will be the same again...

That is a big downside, the question may be how often it happens in practice. If a few variations don't work, I'd personally try changing the variation strength or the sampler or the steps or the prompt or a combination.

[Any-Winter-4079]

Exactly. I think the (or a) solution is my comment just above.

About how often, I don't know, but I would expect to want to use the same strength often enough to be a bit of a pain otherwise (e.g. imagine needing to set new strength all the time, because with used strength values we get old images -and will we remember all those values used? :)

[lstein]

I've fixed it insofar as the pretty-printed dream prompt now indicates the correct seed used to generate the variations, regardless of whether it was fixed in the original dream prompt or assigned automatically. I've been using seed values greater than zero, so the -S0 behavior is not an issue (should this be caught as an assertion error, or is it a feature?)

I can revert the call to randomseed() so that -SXXXX -vY.Y never produce the same series of images. However, this behavior is different from what you get without -v, when -nXX produces the same series of images when given a fixed seed.

My current issue is that when specifying the seed and variant produced by the pretty-printed prompt, I'm not getting the expected variant image back. This illustrates the problem:

dream> a colorful fire hydrant -n3 -v0.15
Outputs:
[1] outputs/img-samples/000011.2529144439.png: "a colorful fire hydrant" -s 50 -W 512 -H 512 -C 7.5 -A k_lms -S 2529144439 -v 0.15 -V 2529144439:0.15
[2] outputs/img-samples/000011.1387204088.png: "a colorful fire hydrant" -s 50 -W 512 -H 512 -C 7.5 -A k_lms -S 2529144439 -v 0.15 -V 1387204088:0.15
[3] outputs/img-samples/000011.1567715846.png: "a colorful fire hydrant" -s 50 -W 512 -H 512 -C 7.5 -A k_lms -S 2529144439 -v 0.15 -V 1567715846:0.15

Say I want to regenerate variant [2], so I cut and paste the corresponding output prompt line:

dream> "a colorful fire hydrant" -s 50 -W 512 -H 512 -C 7.5 -A k_lms -v0.15 -S 2529144439 -V 1387204088:0.15

Nice try, but no match!

I suspect that either:

1. The base seed is not being chosen correctly. Currently it is the seed passed to the image callback for the very first variant in the series.
2. The seed in -V is not correct. This is the seed that is passed back to the image callback.
3. The variation amount in -V is not correct, at least when trying to regenerate a single variant image. However, setting this manually to 0.0 doesn't fix the issue.

I also thought that the variation_amount option (-v) should be set to zero, but this doesn't help.

So at this point I'm going to stop trying to debug the problem and revert the random seed call in get_noise(). I did do some cleanup in dream.py regarding how the pretty-printed prompts are formatted, and I'm committing this to development. I leave it to later (or someone else) to implement @Any-Winter-4079 's proposed solution.

[bakkot]

The base seed is not being chosen correctly. Currently it is the seed passed to the image callback for the very first variant in the series.

That's definitely a problem. It used to be the original seed. (Variations have two seeds, which are interpolated between.) So you would never have the same value in -S as -V in the output from a command with -v.

That must have gotten broken at some point.

I also thought that the variation_amount option (-v) should be set to zero, but this doesn't help.

That is also a problem, independent of the previous thing. -v should not be specified at all in the output from the variation command. -v should be treated more like -n: it is used to determine the parameters for the output images (specifically, it is used to determine -V), rather than being a parameter itself.

That also must have gotten broken at some point; when I initially introduced this feature -v would not appear in the output at all (which was the correct behavior).

[Any-Winter-4079]

I must have an older version because to me
a colorful fire hydrant -n3 -v0.15 produces

[1] outputs/img-samples/001083.944962109.png: "a colorful fire hydrant" -s50 -W512 -H512 -C7.5 -Ak_lms -V944962109:0.15 -SNone
[2] outputs/img-samples/001083.3357559079.png: "a colorful fire hydrant" -s50 -W512 -H512 -C7.5 -Ak_lms -V3357559079:0.15 -SNone
[3] outputs/img-samples/001083.561104318.png: "a colorful fire hydrant" -s50 -W512 -H512 -C7.5 -Ak_lms -V561104318:0.15 -SNone

with no -v and -SNone
Anyway, my fix seems to work on my end, but since there seem to be new changes, it might be a better idea that @bakkot (or the person that has been working on it lately) looks at the correct/expected behavior first.

[psychedelicious]

@bakkot has described the expected behavior, my understanding of what should happen was accurate

• Using -v > 0 always makes a variation, non-deterministic
• Using -V with -v generates a variation based on the seed-weight pairs, non-deterministic
• Using -V without -v generates an image based on the seed-weight pairs, determinisitic
• The output parameters for result images, if re-run, always produces the same image, at least on that machine. No parameters like iterations or -v should be in that output, because they intentionally do non-deterministic things.

Questions:

• Is -S a special case of -V? Is -S 123 === -V 123:1 or something like that? If so, do we need two parameters?
• What happens when we specify -S and -V?
• If variations have two seeds which are interpolated between, why aren't they both represented in -V as this is a special handling of seeds, where -S indicates the normal handling?

Finally - maybe I have missed a revelation while reading through the discussion here that explains this - I have the same issue as originally reported with -S 1 which I assume python considers truth-y:

REPL command 1: test -s 5 -S 1 -v 0.1
REPL output 1: [2] outputs/img-samples/000040.4016874652.png: "test -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S 4016874652 -n 1 -f 0.75 -V 4016874652:0.1
Image 1:

REPL command 2: test -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S 4016874652 -n 1 -f 0.75 -V 4016874652:0.1 (same as output 1)
REPL output 2: [2] outputs/img-samples/000040.4016874652.png: "test -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S 4016874652 -n 1 -f 0.75 -V 4016874652:0.1 (same as REPL output 1 and command 2)
Image 2:

Edit: to be clear, command REPL output 1 produces a reproducible image, it is the same every time. Also, you can remove -V <seed,weight> from the command and the image doesn't change.

[bakkot]

Is -S a special case of -V? Is -S 123 === -V 123:1 or something like that? If so, do we need two parameters?

Not exactly. -S specifies a starting point, and then each -V pair specifies another point and a distance towards that point.

So -S 42 -V 123:0.1 says "start at the noise given by seed 42, then move 10% of the way towards the noise given by seed 123".

As a special case, if you specify :1, that will mean "move all the way towards that point", so -S 456 -V 123:1 will work just like -S 123.

It's probably possible to combine them into a single parameter in some fashion, but it's not really obvious to me that there is a clear way to do that.

What happens when we specify -S and -V?

Hopefully the above answered that question. You are pretty much always going to want to specify -S whenever you specify -V.

If variations have two seeds which are interpolated between, why aren't they both represented in -V as this is a special handling of seeds, where -S indicates the normal handling?

Because the first seed really is special: are starting at that seed, not moving towards it.

If you only have two seeds, yes, you could regard this as being just a weighted average of two seeds, and the order doesn't matter. But once you start getting into variations-of-variations - i.e., when you have more than one value in -V - they're no longer equivalent: starting at point A, moving 50% of the way towards point B, then moving 50% of the way to point C is not the same as starting at point C, moving 50% of the way to point B, then moving 50% of the way to point A.

Finally - maybe I have missed a revelation while reading through the discussion here that explains this - I have the same issue as originally reported with -S 1 which I assume python considers truth-y:

Yeah, there's a separate bug introduced sometime in the last week or two caused by the initial seed getting lost on the way to getting printed. You can tell because the fist line printed in the output has the same seed value in -S and -V, which should never happen.

[psychedelicious]

Thank you for the very clear explanation.

One point I am confused by:

So -S 42 -V 123:0.1 says "start at the noise given by seed 42, then move 10% of the way towards the noise given by seed 123".

As a special case, if you specify :1, that will mean "move all the way towards that point", so -S 456 -V 123:1 will work just like -S 456.

If -S 42 -V 123:0.1 says "start at 42 and go 10% of the way to 123", I would have guessed that -S 42 -V 123:1 says "start at 42 and go all the way to 123" instead of "start at 42 and move all the way back to 42"...

[Any-Winter-4079]

Not to get in the middle of the discussion (I'm reading intently) but what files are involved?

[Any-Winter-4079]

B/c when I try
test -s 5 -S 1 -v 0.1

and ouputs
"test" -s5 -W512 -H512 -C7.5 -Ak_lms -V3422978809:0.1 -S1
which re-entered

Either I have a previous version of the file that just works (and if so, I can share it), or it's something related to M1. But here it works.

[bakkot]

If -S 42 -V 123:0.1 says "start at 42 and go 10% of the way to 123", I would have guessed that -S 42 -V 123:1 says "start at 42 and go all the way to 123" instead of "start at 42 and move all the way back to 42"...

Whoops, sorry, I had a typo in my example (now fixed). It does indeed mean "go all the way to 123".

[bakkot]

Either I have a previous version of the file that just works

Yeah, I think this broke very recently (and probably only on the development branch), so I'm not surprised if you have an older version which still works correctly. Unfortunately I'm way behind on everything, so I can't tell you exactly which file the issue is in.

[lstein]

The base seed is not being chosen correctly. Currently it is the seed passed to the image callback for the very first variant in the series.

That's definitely a problem. It used to be the original seed. (Variations have two seeds, which are interpolated between.) So you would never have the same value in -S as -V in the output from a command with -v.

That must have gotten broken at some point.

I also thought that the variation_amount option (-v) should be set to zero, but this doesn't help.

That is also a problem, independent of the previous thing. -v should not be specified at all in the output from the variation command. -v should be treated more like -n: it is used to determine the parameters for the output images (specifically, it is used to determine -V), rather than being a parameter itself.

That also must have gotten broken at some point; when I initially introduced this feature -v would not appear in the output at all (which was the correct behavior).

At least some of this has resulted from my refactoring and simplifying how the dream command is reconstructed for display on the screen and in the PNG file. I did not understand the nuances of the variation syntax and so sorry to break your code. I'm just getting back to this conversation and will read through to the end before making any more comments.

[lstein]

Ok, I've read through and understand what needs to be done to fix this. I just have to track and use the very original seed for the -S argument. Thanks for a great discussion, and I really appreciate @bakkot 's lucid explanation of the expected behavior and what -V really means.

[lstein]

Should be working now. Fix committed to development.