Finetuning with Artifacts

[wangqiuwen1006]

During the finetuning process, I noticed that the old version of v2 model might produce some grid-like artifacts, whereas the updated v2 version does not have these artifacts. Could you please explain what specifically might be causing this phenomenon?

[lpiccinelli-eth]

The grid artifacts are due to positional encoding dominating the actual features.
Typically it happens in low signal-to-noise regions, like quite texture-less or far away regions.
What we changed is how we compute the scale and shift invariant region during training, namely, we compute the statistics only on valid regions (not on the padded regions that appear due to resizing augmentations), this might have pushed the network to ignore the padded regions, which are like the mentioned low SNR regions that usually present those kind of artifacts.