Robust Gaussian Splatting For Snapshot Compressive Imaging via Large Vision Model Priors
Yanming Yang,
Chenxi Song,
Ping Wang,
Xin Yuan,
Chi Zhang*
* Corresponding author
Overview · Installation · Quick Start · Data Layout · Evaluation
This repository contains the official implementation of GS2CI. Its primary reconstruction combines measurement-derived 3D VFM initialization with SCI-aware Gaussian optimization and jointly refines the Gaussian scene and camera poses. Opacity-Guided Splitting and Growth Regulation (OSGR) stabilizes densification under ambiguous SCI supervision. After coarse-stage convergence, an optional auxiliary stage refines the Gaussian representation using pseudo-view targets generated by a frozen 2D VFM at synthesized viewpoints, while keeping the camera poses fixed.
The pipeline has three steps:
- Form or load an SCI measurement and its per-view masks.
- Construct geometry-oriented proxy views and use VGGT to estimate the initial cameras and point cloud.
- Jointly optimize the Gaussian scene and camera poses with SCI measurement consistency and OSGR in the coarse stage, followed by optional refinement of the Gaussian representation with fixed camera poses.
GS2CI is implemented as Nerfstudio model, data-manager, trainer, and camera-optimizer subclasses. Nerfstudio and gsplat are installed as upstream dependencies; their source trees are not vendored here.
The code is tested with Python 3.10, PyTorch 2.1.2, Nerfstudio 1.1.5, gsplat 1.5.0, and CUDA 11.8.
conda create -n gs2ci python=3.10 -y
conda activate gs2ci
pip install torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu118
pip install -e .
pip install --no-deps --force-reinstall "gsplat==1.5.0" \
--index-url https://docs.gsplat.studio/whl/pt21cu118Nerfstudio 1.1.5 declares an exact gsplat 1.4.0 dependency, while this implementation uses gsplat 1.5.0. The final command intentionally upgrades that runtime dependency without vendoring either project.
Install optional extras before the final gsplat override. For example, a full evaluation-and-fine environment uses:
pip install -e ".[eval,fine]"
pip install --no-deps --force-reinstall "gsplat==1.5.0" \
--index-url https://docs.gsplat.studio/whl/pt21cu118The second command must remain last because Nerfstudio 1.1.5 otherwise resolves
gsplat back to 1.4.0. The fine dependency versions are pinned to the
DiFix3D-compatible Diffusers/PEFT/Hugging Face stack. DiFix weights are loaded
from nvidia/difix_ref by default; pass --difix-model /path/to/difix_ref to
use a local checkpoint. The final command installs gsplat's official
PyTorch-2.1/CUDA-11.8 binary; the training wrapper validates that packaged CUDA
backend before training and rejects an incompatible shared-cache extension.
Install VGGT in a separate environment. First pin a clean checkout, then install that commit's official VGGT dependencies, including the COLMAP demo dependencies. The initializer is locked to PyTorch 2.1.2, torchvision 0.16.2, and the CUDA 11.8 PyTorch wheels used to produce the released initialization. Run both pinning commands below last; do not reinstall the official requirements afterward. No VGGT source modification is required.
conda create -n gs2ci-vggt python=3.10 -y
conda activate gs2ci-vggt
git -C ../vggt checkout 44b3afbd1869d8bde4894dd8ea1e293112dd5eba
git -C ../vggt status --short # must be empty
python -m pip install -r ../vggt/requirements.txt
python -m pip install -r ../vggt/requirements_demo.txt
python -m pip install \
"torch==2.1.2" "torchvision==0.16.2" \
--index-url https://download.pytorch.org/whl/cu118
python -m pip install --force-reinstall --no-deps \
"numpy==1.26.4" "opencv-python==4.8.0.76" "plyfile==1.1.3"
(cd ../vggt && python -c "import cv2, demo_colmap, numpy, torch, torchvision; print(torch.__version__, torchvision.__version__, torch.version.cuda, torch.backends.cudnn.version(), numpy.__version__, cv2.__version__)")For synthetic SCI data from clean frames:
python scripts/prepare_sci_data.py \
--frames data/source/frames \
--masks data/source/masks.npy \
--output data/example \
--copy-frames-as-gtFor a captured measurement, replace --frames with --measurement data/source/meas.npy. The script prepares the measurement, masks, and geometry-oriented VGGT proxy views under data/example.
Before VGGT estimation, input preparation constructs geometry-oriented proxy views from the SCI measurement and the actual coding masks. The default --vggt-proxy auto selects one of two construction branches solely according to the mean measurement multiplicity (the mean active-mask count per pixel), using the released threshold of 4:
- Energy-Normalized Initialization (ENI) for low-multiplicity acquisitions (mean active-mask count below 4). It normalizes the aggregate exposure and completes each mask-supported proxy on the original image grid.
- Adaptive Mask-Decoding Initialization (AMDI) for high-multiplicity acquisitions (mean active-mask count at least 4). It estimates local per-view proxy contributions from coding-pattern diversity through adaptive, regularized least squares.
Both branches produce a complete proxy sequence for the same frozen VGGT model. These geometry-oriented proxies are used only for initialization; they are neither recovered latent frames nor photometric supervision. run_vggt.py consumes the resulting images, while --expected-proxy-mode auto verifies the automatically prepared mode before estimation.
python scripts/run_vggt.py \
--data data/example \
--vggt-root ../vggt \
--python /path/to/vggt-env/bin/python \
--expected-proxy-mode auto \
--gpu 0Before launching VGGT, the command checks the recorded mode against the actual masks and verifies the measurement, mask, and prepared-image hashes. It writes the selected COLMAP reconstruction to the scene's sparse/ directory. Pass --model /path/to/VGGT-1B and --tracker-model /path/to/vggsfm_v2_tracker.pt to use local weights.
Run the 20,000-iteration coarse stage:
python scripts/train_gs2ci.py \
--data data/example \
--output-dir outputs \
--experiment-name example_coarse \
--stage coarse \
--gpu 0Continue for the 3,000-iteration fine stage from the generated checkpoint directory:
python scripts/train_gs2ci.py \
--data data/example \
--output-dir outputs \
--experiment-name example_fine \
--stage fine \
--load-dir outputs/example_coarse/gs2ci/<run>/nerfstudio_models \
--gpu 0Use --vis viewer for the GS2CI-aware Viser viewer. The wrapper prints the complete ns-train command before execution so every run is auditable.
The prepared SCI benchmark data for the six main cases—Airplants, Cozy2room, Factory, Hotdog, Tanabata, and Vender—are obtained from the SCINeRF release. The datasets are not redistributed in this repository; please follow SCINeRF's instructions to download them, then arrange each scene using the layout below.
data/example/
|-- images/meas.npy
|-- masks/mask.npy
|-- scene_metadata.json
|-- sparse/{cameras,images,points3D}.bin
|-- vggt/images/*.png
`-- gt/ # optional, evaluation only
The measurement must be the SCI sum of masked latent views, not its per-pixel average. The ENI/AMDI outputs are geometry-oriented VGGT proxy views rather than latent-frame reconstructions or photometric targets. The mask stack must have shape (N,H,W), match the measurement resolution, and pass the selected proxy constructor's validity checks.
Install evaluation dependencies and re-apply the required gsplat override:
pip install -e ".[eval]"
pip install --no-deps --force-reinstall "gsplat==1.5.0" \
--index-url https://docs.gsplat.studio/whl/pt21cu118Then run:
python scripts/evaluate.py --gt data/example/gt --pred /path/to/rendered/base_viewsThis implementation builds on Nerfstudio, gsplat, VGGT, DiFix3D+, and SCINeRF. We thank the authors of these projects for making their work publicly available. Please follow their licenses and citation requirements when using the corresponding components or checkpoints.
Original GS2CI code is released under the Apache License 2.0. The optional fine-stage DiFix3D component and other third-party software remain subject to their upstream terms; see THIRD_PARTY_NOTICES and LICENSES/DIFIX3D-NVIDIA.txt for details. Model weights are not distributed in this repository.
If you find this work useful, please cite our paper:
@misc{yang2026gs2cirobustgaussiansplatting,
title = {GS$^{2}$CI: Robust Gaussian Splatting For Snapshot Compressive Imaging via Large Vision Model Priors},
author = {Yanming Yang and Chenxi Song and Ping Wang and Xin Yuan and Chi Zhang},
year = {2026},
eprint = {2608.13502},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2608.13502}
}