Jonas Kulhanek · Songyou Peng · Zuzana Kukelova · Marc Pollefeys · Torsten Sattler
NeurIPS 2024
We introduce WildGaussians, a novel approach to handle occlusions and appearance changes with 3DGS. By leveraging robust DINO features and integrating an appearance modeling module within 3DGS, our method achieves state-of-the-art results. We demonstrate that WildGaussians matches the real-time rendering speed of 3DGS while surpassing both 3DGS and NeRF baselines in handling in-the-wild data, all within a simple architectural framework.
💫 NEWS: WildGaussians is now integrated into NerfBaselines. Checkout the results here: https://nerfbaselines.github.io/phototourism
We offer two ways to install the method: local installation (through a conda environment) and installation from NerfBaselines, where multiple backends (conda,docker,apptainer) can be selected.
WildGaussians is already included in the newest release of NerfBaselines. If you don't have NerfBaselines installed, you can install it by running:
conda create -n nb python=3.11
conda activate nb
pip install nerfbaselines>=1.2.0Clone the repository and create a python == 3.11 Anaconda environment with CUDA toolkit 11.8 installed using
git clone [email protected]:jkulhanek/wild-gaussians.git
cd wild-gaussians
conda create -y -n wg python=3.11
conda activate wg
conda install -y --override-channels -c nvidia/label/cuda-11.8.0 cuda-toolkit
conda env config vars set NERFBASELINES_BACKEND=python
pip install --upgrade pip
pip install -r requirements.txt
pip install nerfbaselines>=1.2.0
pip install -e ./submodules/diff-gaussian-rasterization ./submodules/simple-knn
pip install -e .The checkpoints and predictions are available online at https://huggingface.co/jkulhanek/wild-gaussians/tree/main:
- Photo Tourism/Trevi Fountain
- Photo Tourism/Sacre Coeur
- Photo Tourism/Brandenburg Gate
- NeRF On-the-go/Fountain
- NeRF On-the-go/Mountain
- NeRF On-the-go/Corner
- NeRF On-the-go/Patio
- NeRF On-the-go/Spot
- NeRF On-the-go/Patio High
In order to train/evaluate on the NeRF On-the-go dataset, please download the undistorted version from the following link: https://huggingface.co/datasets/jkulhanek/nerfonthego-undistorted/tree/main
To start the viewer and explore the trained models, run one of the following:
# Photo Tourism
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/trevi-fountain.zip/checkpoint --data external://phototourism/trevi-fountain
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/sacre-coeur.zip/checkpoint --data external://phototourism/sacre-coeur
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/brandenburg-gate.zip/checkpoint --data external://phototourism/brandenburg-gate
# NeRF on-the-go
# NOTE: Optionally attach `--data {path to data}` to load the dataset (recommended)
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego-undistorted/fountain.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego-undistorted/mountain.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego-undistorted/corner.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego-undistorted/patio.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego-undistorted/spot.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego-undistorted/patio-high.zip/checkpointTo start the training on the Photo Tourism dataset, run one of following commands:
# Photo Tourism
nerfbaselines train --method wild-gaussians --data external://phototourism/trevi-fountain
nerfbaselines train --method wild-gaussians --data external://phototourism/sacre-coeur
nerfbaselines train --method wild-gaussians --data external://phototourism/brandenburg-gate
# NeRF on-the-go
nerfbaselines train --method wild-gaussians --data {path to data}The training will also generate output artifacts containing the test set predictions, checkpoint, and tensorboard logs.
To render a video on a trajectory (e.g., generated from the interactive viewer), run:
nerfbaselines render --checkpoint {checkpoint} --trajectory {trajectory file}There are several concurrent works that also aim to extend 3DGS to handle in-the-wild data:
- Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections
- Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections
- SpotlessSplats: Ignoring Distractors in 3D Gaussian Splatting
- SWAG: Splatting in the Wild images with Appearance-conditioned Gaussians
- WE-GS: An In-the-wild Efficient 3D Gaussian Representation for Unconstrained Photo Collections
We would like to thank Weining Ren for his help with the NeRF On-the-go dataset and code and Tobias Fischer and Xi Wang for fruitful discussions. This work was supported by the Czech Science Foundation (GAČR) EXPRO (grant no. 23-07973X) and by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254). Jonas Kulhanek acknowledges travel support from the European Union’s Horizon 2020 research and innovation programme under ELISE Grant Agreement No 951847. The renderer is built on 3DGS and Mip-Splatting. Please follow the license of 3DGS and Mip-Splatting. We thank all the authors for their great work and released code.
If you find our code or paper useful, please cite:
@article{kulhanek2024wildgaussians,
title={WildGaussians: 3D Gaussian Splatting in the Wild},
author={Kulhanek, Jonas and Peng, Songyou and Kukelova, Zuzana and Pollefeys, Marc and Sattler, Torsten},
journal={arXiv},
year={2024}
}