Add Deep Blending results, full evaluation script, and per-scene results #388
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a new evaluation script for the Deep Blending dataset, a new evaluation script to eval all datasets with different numbers of Gaussians and write the per scene results in csvs for the 3dgs.zip survey.
To this end the 2 existing evaluation scripts are modified to accept arguments.
The PSNR results on the Deep Blending dataset (dark brown) seem strange as they decrease with more Gaussians, but the SSIM and LPIPS results seem plausible. Any ideas on how to explain this or if it could be a bug related to the data loading are welcome @jefequien.
The dataset can be downloaded with the updated download script in #385.
When skipping through the renderings, compress_0022.png of scene playroom (1M Gaussians) was the only really bad results that i noticed
However this is probably only an outlier due to lightening because it looks very different in compress_0018.png in a slightly different angle.
So maybe it is caused by the data itself that this approach leads to decreasing PSNR results with more Gaussians..