Configs for llff dataset

[JosTheBossX]

Hello,

I have 2 questions:

1. I have seen that I run out of CUDA memory everytime I try to run this code on Ficus dataset. I am running them on A100s (80GB). Do we need to set different config for it?
2. So for llff, dataset I tried to use the config from torch-ngp, but it doesn't give me consistent results. For eg, for blender dataset, all the exp get trained for 100 epochs, but for LLFF it seems to go over that, and I didn't see any perfect number for all the LLFF dataset.

So can you please let me know what config/arguments did you use for the ficus and LLFF datasets?

[qsong2001]

Hi,

Thank you for your attention to our work.

For Q1.

You could try down-sample the Ficus dataset, using fp16 mode and deactivating the preload.

In our paper, we experiment with 3 scenes from LLFF with a 4x down-sample, from NeRF-syn with 2x down-sample;
3 rendering layers are used for message embedding.

Scenes	PSNR	SSIM	LPIPS	Bit Acc.
Lego	31.32568	0.954157	0.029384	0.919583
Drum	24.594726	0.905858	0.077093	0.894063
Chair	31.858801	0.957991	0.038413	0.969792
{lego, drums,chair }	29.25973567	0.939335333	0.04829667	0.92781267

Scenes	PSNR	SSIM	LPIPS	Bit Acc.
Horn	24.045519	0.791919	0.146275	0.989583
Fern	27.603946	0.877897	0.054258	0.9375
Fortress	28.801754	0.900996	0.04954	0.982639
{flower,fern,fortress}	26.817073	0.856937333	0.08335767	0.96990733

For Q2.

Instant-NGP performs not very well on the LLFF dataset due to the colormap (NVlabs/instant-ngp#367). Unlike the NeRF-synthetic data, we also find that watermarks are easier to embed as the change of viewpoint is limited. In such conditions, using a single-scale rendering can also effectively embed watermarks, and the image quality will be higher in this setting. You may also try to reduce the loss item of message embedding.

Thanks.

[JosTheBossX]

Hey thanks for your reply.

Yes I just tried to downsample it by 4, it is working now.
I haven't really worked with ngp, but isn't it weird, all of the syn datasets were working fine, but only ficus has the issue of OOM. All the other datasets I tried to train, I was successful to train them with 2 downsampling (and even the resolutions are same for all syn data).

Yeah it doesn't perform really well on LLFF for quality (bit ACC are fine). I found that at least other LLFF dataset reconstruct well, but flower and leaves don't even get a good reconstruction as seen in this figs:

As you mentioned that you used 3 LLFF datasets, and one of them is flower; how did u manage to get good reconstruction?

[qsong2001]

We used the transformation code of ngp_pl to deal with the llff dataset.
Please refer to this issue: NVlabs/instant-ngp#367.
Thanks and sorry for the late reply.