Awesome Inverse Graphics

This is a curated list of resources and papers related to inverse graphics.

This list will most certainly miss things, so please feel free to add a pull request or contact me directly at [email protected].

Inverse graphics is a super exciting field, and hope this awesome list gets you as excited about some of these research directions as I am!

What is inverse graphics?

According to Yildrim et al. in their 2015 Cog Sci paper

The analysis-by-synthesis or “vision as inverse graphics” approach presents one way to think about how vision can be so rich in its content. The perceptual system models the generative processes by which natural scenes are constructed, as well as the process by which images are formed from scenes; this is a mechanism for the hypothetical “synthesis” of natural images, in the style of computer graphics. Perception (or “analysis”) is then the search for or inference to the best explanation of an observed image in terms of this synthesis model: What would have been the most likely underlying scene that could have produced this image?

Papers

The Helmholtz Machine (Dayan et al., 1995)

Generative models for discovering sparse distributed representations (Hinton & Ghahramani, 1997)

Instantiating Deformable Models with a Neural Net (Williams, Revow, & Hinton, 1997)

Object Perception as Bayesian Inference (Kersten, Mamassian, & Yuille, 2004)

Vision as Bayesian Inference: Analysis by Synthesis? (Yuille & Kersten, 2006)

Analysis-by-Synthesis by Learning to Invert Generative Black Boxes (Nair, Susskind, & Hinton, 2008)

Analysis by Snythesis: A (Re-)Emerging Program of Research for Language and Vision (See Section 7) (Bever & Poeppel, 2010)

Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs (Mansinghka et al., 2013)

Whatever next? Predictive brains, situated agets, and the future of cognitive science (Clark, 2013)

The Informed Sampler: A Discriminative Approach to Bayesian Inference in Generative Computer Vision Models (Jampani et al., 2014)

Inverse Graphics with Probabilistc CAD Models (Kulkarni et al., 2014)

Efficient analysis-by-snythesis in vision: A computational framework, behavioral tests, and comparison with neural representations (Yildirim et al., 2015)

Picture: A Probabilistic Programming Language for Scene Perception (Kulkarni et al., 2015)

Deep Convolutional Inverse Graphics Network (Kulkarni et al., 2015)

Galileo: Perceiving physical object properties by integrating a physics engine with deep learning (Wu et al., 2015)

Perceiving Fully Occluded Objects via Physical Simulation (Yildirim, Siegel, & Tenenbaum, 2016)

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models (Eslami et al., 2016)

Single Image 3D Interpreter Network (Wu et al., 2016)

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction (Choy et al., 2016)

Overcoming Occlusion with Inverse Graphics (Moreno et al., 2016)

Neural Scene De-Rendering (Wu, Tenenbaum, & Kohli, 2017)

Vision-as-Inverse-Graphics: Obtaining a Rich 3D Explanation of a Scene from a Single Image (Romaszko et al., 2017)

Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision (Tung et al., 2017)

Learning to See Physics via Visual De-animation (Wu et al., 2017)

3D-Aware Scene Manipulation via Inverse Graphics (Yao et al., 2018)

Visual Object Networks: Image Generation with Disentangled 3D Representation (Zhu et al., 2018)

Neurocomputational Modeling of Human Physical Scene Understanding (Yildirim et al., 2018)

Learning to Infer Graphics Programs from Hand-Drawn Images (Ellis et al., 2018)

3D-RCNN: Instance-level 3D Object Reconstruction via Render-and-Compare (Kundu, Li, & Rehg, 2018)

Learning to Reconstruct Shapes from Unseen Classes (Zhang et al., 2018)

An integrative computational architecture for object-driven cortex (Yildirim et al., 2019)

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (Sitzmann, Zollhofer, & Wetzstein, 2019)

InverseRenderNet: Learning single image inverse rendering (Yu & Smith, 2019)

IGE-Net: Inverse Graphics Energy Networks for Human Pose Estimation and Single-View Reconstruction (Jack et al., 2019)

Write, Execute, Assess: Program Synthesis with a REPL (Ellis et al., 2019)

A Neural-Symbolic Architecture for Inverse Graphics Improved by Lifelong Meta-Learning (Kissner & Mayer, 2019)

Efficient inverse graphics in biological face professing (Yildirim et al., 2020)

3D Morphable Face Models - Past, Present and Future (Egger et al., 2020)

Building 3D Morphable Models from a Single Scan (Sutherland, Egger, & Tenenbaum, 2020)

Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering (Zhang et al., 2020)

Deep Nets: What have They Ever Done for Vision? (Yuille & Liu, 2020)

Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data (Lunz et al., 2020)

Physics-as-Inverse Graphics: Unsupervised Physical Parameter Estimation from Video (Jaques, Burke, & Hospedales, 2020)

DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning (Ellis et al., 2020)

SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images (Lin, Wang, & Lucey, 2020)

Cycle-Consistent Generative Rendering for 2D-3D Modality Transition (Aumentado-Armstrong et al., 2020)

General Resources

What is an Inverse Problem?

Curated List of 3D Morphable Models

Pattern Synthesis (Grenander, 1976)

A Stochastic Grammar of Images (Zhu & Momford, 2007)

Vision as Inverse Graphics for Detailed Scene Understanding (Moreno, 2019)

Machine Learning for Machine Vision as Inverse Graphics Course (Bonner, 2020)

Related Talks + Videos

Does the Brain Do Inverse Graphics? (Hinton, 2015)

Probabilistic Programming for Augmented Intelligence (Mansinghka, 2016)

Computational Models of Cognition (Tenenbaum, 2018)

Inverse Programming for Deeper AI (Tavares TWIML podcast interview, 2018)

Solving Inverse Graphics Problems In-the-Wild (Kanazawa, 2018)

Learning to Solve Inverse Problems in Imaging (Willet, 2019)

Reverse-engineerring Core Common Sense (Tenenbaum, 2020)

Compositional Models and Occlusion (Yuille, 2020)

AI for 3D Content Creation (Fidler, 2020)

Implicit Neural Representations (Sitzmann, 2020)

Integrating Learning and Graphics for 3D Scene Understanding (Wu, 2020)

Common Sense Reasoning through Causal Probabilistic Programming (Tavares, 2020)

*Inspired by Awesome Implicit Neural Representations and Awesome Computer Vision. And a huge thanks to Bernhard Egger for help with the list, as well!