Added a reflectance padding option from leongatys/NeuralImageSynthesis

[ProGamerGov]

The new command is used similar to the -normalize_gradients command, with: -reflectance. By default, it is set to false.

The related issue can be found here: #376

[htoyryla]

See #376 (comment) and the continuation about a potential problem in this modification.

[ProGamerGov]

This new feature should be working correctly now.

[htoyryla]

This pull request is part of an experimental attempt to extend neural-style towards "Controlling Perceptual Factors in Neural Style Transfer" (see #376). I am not convinced of the value of changing the existing neural-style in this way; I would rather see parallel projects in which the basic neural-style code is developed to different directions with new functionality (like so many have already done).

My main reason for this is that neural-style is a very good starting point to explore implementing style transfer in lua. The code is as compact, clear and simple as it can be. Why not keep it that way. Adding experimental features into it will make it more difficult to understand and re-use, more error prone to develop further and even use. I have done numerous experiments based on this code, but preferred not to offer them into the main neural-style branch.

[VaKonS]

@htoyryla, this particular "padding" option seems to be useful with small images – it removes dark spots near the border.
(And adds some noise, see bottom edge of second image. Didn't understand why yet).
Without and with padding:

[htoyryla]

Note that this is not my decision anyway, just an opinion. I can understand the addition of a set of padding options, if we see that padding really makes a difference. And perhaps it does... fast-neural-style does offer many padding options using customized padding layers, although it looks like they are only used in the feed-forward version, slow-neural-style.lua does not use them.

I would prefer a new parameter -padding, which could take several values:

• the default being 'default' meaning "leave padding to the convolution layer".
• 'reflect' meaning the new behaviour here (the word 'reflectance' sounds strange to me, nn uses 'reflection')

nn further offers replication padding, which could be added later (although there is little difference between reflection and replication when padding with one pixel.

[ProGamerGov]

@htoyryla Changing the -reflectance parameter to -padding should be easy enough, and if future padding options arise, a new command won't be needed.

Edit: The parameter is now known as -padding and currently takes either default or reflect, with default being the default option.

[VaKonS]

@htoyryla, the error seems to accumulate, and difference in 1-pixel border makes noticeable changes after relatively high number of iterations (the above images are made with 300 iterations).

"Replication" variant makes difference with small details near the edges (reflection and replication):

"Reflection" eats those details, "replication" can stretch some elements to the border (see red ribbon on the left side). So both options make sense.

[htoyryla]

Really, I cannot understand how there can be a difference if you add one row or column by reflection or replication. Both add a copy of the row/column on the edge. Only when you pad with several rows/columns does the padding start to differ. Did you use the same seed for both runs to exclude the randomness? Or did you use some model which actually pads with more than one row/column? Anyway, I never said that reflect and replicate would not both be valid options, only that they result in identical padding when padding with one row and column. What I said was that zero is superfluous and pad is meaningless. Lähetetty iPadista

…

Valentin ***@***.***> kirjoitti 15.2.2017 kello 15.53: @htoyryla, the error seems to accumulate, and difference in 1-pixel border makes noticeable changes after relatively high number of iterations (the above images are made with 300 iterations). "Replication" variant makes difference with small details near the edges (reflection and replication): "Reflection" eats those details, "replication" can stretch some elements to the border (see red ribbon on the left side). So both options make sense. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[htoyryla]

Or maybe I have misunderstood what reflection and replication means. I need to test both to see exactly what they add to a tensor.

[VaKonS]

@htoyryla, all three images are made with same settings, the only difference is padding option – default, reflection and replication.

The seed number is fixed too.

[htoyryla]

I was wrong... misunderstood how reflection works when it is stated that the values are reflected of the input boundary. They are actually reflected over the edge row, not reflections of the edge row. So if the row ends with values ... 2 4 5, replication padding gives ... 2 4 5 5 while reflection gives ... 2 4 5 4 ( so that the mirror is the edge row, not the edge as I had understood). Like this: th> t = torch.Tensor(1,4,4) [0.0001s] th> t (1,.,.) = 6.9249e-310 6.9249e-310 0.0000e+00 0.0000e+00 2.4414e-152 1.3368e-152 3.0544e-67 6.0133e-154 4.0324e+175 1.7872e+209 3.2489e-86 2.8553e-109 3.9064e+180 8.7210e+223 1.1723e+214 9.8168e+252 [torch.DoubleTensor of size 1x4x4] [0.0002s] th> nn.SpatialReplicationPadding(1,1,1,1):forward(t) (1,.,.) = 6.9249e-310 6.9249e-310 6.9249e-310 0.0000e+00 0.0000e+00 0.0000e+00 6.9249e-310 6.9249e-310 6.9249e-310 0.0000e+00 0.0000e+00 0.0000e+00 2.4414e-152 2.4414e-152 1.3368e-152 3.0544e-67 6.0133e-154 6.0133e-154 4.0324e+175 4.0324e+175 1.7872e+209 3.2489e-86 2.8553e-109 2.8553e-109 3.9064e+180 3.9064e+180 8.7210e+223 1.1723e+214 9.8168e+252 9.8168e+252 3.9064e+180 3.9064e+180 8.7210e+223 1.1723e+214 9.8168e+252 9.8168e+252 [torch.DoubleTensor of size 1x6x6] [0.0171s] th> nn.SpatialReflectionPadding(1,1,1,1):forward(t) (1,.,.) = 1.3368e-152 2.4414e-152 1.3368e-152 3.0544e-67 6.0133e-154 3.0544e-67 6.9249e-310 6.9249e-310 6.9249e-310 0.0000e+00 0.0000e+00 0.0000e+00 1.3368e-152 2.4414e-152 1.3368e-152 3.0544e-67 6.0133e-154 3.0544e-67 1.7872e+209 4.0324e+175 1.7872e+209 3.2489e-86 2.8553e-109 3.2489e-86 8.7210e+223 3.9064e+180 8.7210e+223 1.1723e+214 9.8168e+252 1.1723e+214 1.7872e+209 4.0324e+175 1.7872e+209 3.2489e-86 2.8553e-109 3.2489e-86 [torch.DoubleTensor of size 1x6x6]

[VaKonS]

@htoyryla, I see the point. I used "vgg_normalised" model and fixed seed number.
And tried several times with different styles and image sizes, so the results should not be accidental.

[htoyryla]

Of course they aren't... the problem for me was why there would be difference. It is clear now, based on how reflection really works. The document is not very clear... the one is replication of the input boundary, the other reflection of the input boundary.

…

Valentin ***@***.***> kirjoitti 15.2.2017 kello 16.55: @htoyryla, I see the point. I used "vgg_normalised" model and fixed seed number. And tried several times with different styles and image sizes, so the results should not be accidental. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[htoyryla]

The error statement is needed, not only to notify the user, but to stop execution because we don't know what exactly should be done. That's why an error() statement is used and not simply print().

[ProGamerGov]

@htoyryla Looking at the other uses of error() in Neural-Style, that makes sense. Thanks for the insight.

[VaKonS]

I was wrong, the noise is not added by padding, it seems to be a feature of some models.
Here is a clean result of one model and noisy of another (default padding / reflection / replication):

More tests in the discussion.