Project Page | Paper | Data
PyTorch implementation of rolling shutter effect correction with NeRF.
USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields
Moyang Li*1,2,
Peng Wang*1,3,
Lingzhe Zhao1,
Bangyan Liao1,3,
Peidong Liu†1,3,
1Westlake University, 2ETH Zurich, 3Zhejiang University
* denotes equal contribution
† denotes corresponding author
in ICLR 2024
USB-NeRF is able to correct rolling shutter distortions and recover accurate camera motion trajectory simultaneously under the framework of NeRF, by modeling the physical image formation process of a rolling shutter camera.
git clone https://github.com/WU-CVGL/USB-NeRF
cd USB-NeRF
pip install -r requirements.txt
You can download the data here.
After acquiring the data, your folder structure should look like
Dataset/
Unreal-RS/
Adornment/
images/
start/
mid/
groundtruth.txt
poses_bounds.npy
BlueRoom/
...
LivingRoom/
...
WhiteRoom/
...
intrinsics.txt
...
images
folder contains captured rolling shutter images. start
and mid
folder contain global shutter images corresponding to the first and middle scanline, respectively. groundtruth.txt
file saves the groundtruth poses, while poses_bounds.npy
file saves the estimated camera poses with rolling shutter images via COLMAP. intrinsics.txt
saves camera intrinsics (fx, fy, cx, cy).
Modify parameters of config file (eg: configs/Unreal-RS/Adornment_CubicSpline.txt
) if needed.
python train_usb_nerf.py --config ./configs/Unreal-RS/Adornment_CubicSpline.txt
After training, you can get global shutter images, optimized camera poses and synthesized novel view images.
If you find this useful, please consider citing our paper:
@article{li2023usb,
title={USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields},
author={Li, Moyang and Wang, Peng and Zhao, Lingzhe and Liao, Bangyan and Liu, Peidong},
journal={arXiv preprint arXiv:2310.02687},
year={2023}
}
The overall framework, metrics computing and camera transformation are derived from nerf-pytorch, CVR and BAD-NeRF respectively. We appreciate the effort of the contributors to these repositories.