Fovea Stacking

Fovea Stacking: Imaging with Dynamic Localized Aberration Correction
Shi Mao, Yogeshwar Nath Mishra, Wolfgang Heidrich
King Abdullah University of Science and Technology (KAUST)
ACM Transactions on Graphics (SIGGRAPH Asia), 2025

Fovea Stacking is a novel imaging paradigm that dynamically controls a deformable phase plate (DPP) to locally compensate for lens aberrations (analogous to the eye's fovea), thus producing regionally corrected images that can be stacked to form an aberration-free composite image.

News

• Aug 2025: Fovea Stacking was accepted as a journal paper at SIGGRAPH Asia 2025.

Environment

conda create -n foveastacking python=3.10
conda activate foveastacking
pip install -r code/requirements.txt 

Quickstart

1. Optimize single DPP patterns for given Region of Interest (ROI)

Initialized as a flat surface without deformation, the DPP is optimized for a given --ROI ([x, y, w, h] defined in relative coordinates).

cd code
python opt_DPP_ROI.py --ROI 0.6 0.6 0.4 0.4 
# Results are saved in `code/results/date-time-opt_DPP_ROI_[0.6,0.6,0.4,0.4]`

2. Optimize multiple DPP patterns to jointly cover the Field of View (FoV)

--n_dpps specifies the number of DPP patterns to be jointly optimized. All DPPs are initialized and optimized together to adaptively specialize in different regions.

opt_DPP_Joint.mp4

cd code
python opt_DPP_joint.py --n_dpps 5
# Results are saved in `code/results/date-time-opt_DPP_joint_5DPPs`

3. Fovea Stacking

Stack multiple captured or simulated images. --result_dir is the folder containing the images, and --img_list specifies the suffix of the image files.

cd code
python fovea_stack.py --result_dir data/camera_captured --img_list *.jpg
# Results are saved inplace in `result_dir/fovea_stack`

Advance Usage

For single or multiple DPP optimization, a --config configuration file can be specified to change the lens settings and optimization parameters. Two example configuration files and corresponding lens files are provided: one for lens settings focused at 60m (hyperfocal) and another for a calibrated 652mm distance.

System Modeling

The DeepLens differentiable optics system operates in a right-handed physical coordinate system, referred to as the Lens Coordinates in the following diagram. The DPP Coordinate is defined locally, with its z-axis pointing towards its deformable surface. For images captured by the sensor, the Image Coordinate is a 2D coordinate in physical units, while the Relative Coordinate normalizes it by the sensor size (and flips the y-axis), and the Pixel Coordinate is scaled by the number of pixels accordingly. Note that the ROI is usually defined in Relative Coordinates.

Hardware Specifications Camera:[GS3-U3-41C6C] Lens:[AC254-050-A] DPP: [Phaseform Delta7]

Citation

Please consider citing our work if you find this codebase useful.

@article{mao2025fovea,
      title     = {Fovea Stacking: Imaging with Dynamic Localized Aberration Correction},
      author    = {Mao, Shi and Mishra, Yogeshwar and Heidrich, Wolfgang},
      journal   = {ACM Transactions on Graphics (TOG)},
      year      = {2025},
      volume    = {44},
      number    = {6},
      articleno = {258},
      month     = {dec},
      doi       = {10.1145/3763278},
      publisher = {ACM}
}

Acknowledgement

The code for differentiable optics in this repository is based on DeepLens.