Why train background area with random colors? / How to mask away training data (e.g. for dynamic object removal)?

[xmk2222]

Thanks for this great work! congratulations!

I'm trying to train nerf models on scenes with dynamic objects, and I noticed the flag random_bg_color, which means training the background area(where alpha channel in RGBA equals zero?) with random colors, I reckon this would make the background area transparent.

So I tried masking out the dynamic objects with 0 alpha value, but the counterpart area on the rendered scene is sometimes black and sometimes gray(partly transparent).

I'm wondering am I doing it right? And why don't you just skip the background area while training instead of training it with random color? Would it be better as it saves calculations?

[Tom94]

Why random background colors?

Hi there, alpha==0 supervises (enforces) transparency, which conflicts with your goal of simply ignoring regions of some of the training images.

Using random colors rather than fixed colors improves this process, because the model can't just predict a solid color. It is forced to learn transparency in order to let the randomized colors "shine through".

How to mask away training data?

This codebase does allows specifying masks for ignoring training pixels, but this is distinct from marking background regions, i.e. serves a different goal.

The format is as follows: for any training image xyz.* with moving objects, you can provide a mask by including a file dynamic_mask_xyz.png in the same folder. This file must be in PNG format, where non-zero pixel values indicate masked-away regions.

(You don't need to include blank masks for training images that don't require any masking.)

I will add this to the FAQ section of the readme

[yoterel]

I carefully read this thread and another (issue_92) regarding usage of the alpha channel in this project. But I still fail to understand why would using random colors as ground truth (where the alpha channel is 0) force the network to learn transparency in that area?

For example, if pixel ij was rendered and has color red this iteration, as a result of integrating over a ray shot from a particular direction, and in the next iteration the same pixel is green, there is no possible explanation for this phenomenon using the parametric form (density, rgb). If the model chose to "learn" density zero along this ray, then it must yield a constant background color (which can only be either green, or red, but not both). This example extends to other pixels that are near ij, or to other pixels that were the result of sampling over a ray that intersects the original.

What am I missing?

I thought that maybe the random color is added to the result of the color output from the network, and not to the training set. Is this true?

[Tom94]

The important piece of information is that the random background color gets composited on both the training image as well as the network prediction. So the network doesn't actually have to explain the color at all -- the color is injected by the training algorithm as a byproduct of the network predicting transparency.

Another way to look at it: whenever the network predicts non-zero density and a different color from the background, the gradients will encourage the network both to update its predicted color and to become more transparent (to let the more-correct background color shine through). By randomizing the colors, any optimization attempts by network to directly predict the color are thwarted (as you say: it has no chance!) and only the parameter updates to transparency survive.

[yoterel]

But in this case, the network can find two plausible solutions: either decreasing the density ("color shines through") or keeping the density intact but predicting an all-black color (and then the sum of all-black and the random color yields the random color as required). If compositing means multiplication, the same explanation applies with all-white. Why would it be forced to choose one over the other? (i.e. transparency isn't enforced)

Also, isn't compositing the random color background in both the training image and the network output equivalent to setting the background of the training image to black? (pixel_ij will want to go to 0 where the background mask is true in both cases)

[Tom94]

Composition depends on transparency. If the volume is fully opaque, none of the background color gets composited, and if it's transparent, all of it gets. The volume rendering equation pretty much becomes alpha blending (with premultiplied alpha) as far as the background color is concerned.

[Harper714]

@Tom94 Why masking out the dynamic objects with 0 alpha cannot make the object transparent? why it only works for the background?

Asking this because I found that if simply masking away training pixels, the rendered area back of the masked object can have artifacts, especially when the masked object is very close to the camera. I guess the reason is that the corresponding feature grid cannot be optimized?