Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition

2020

Conference Paper

avg

Neural rendering techniques promise efficient photo-realistic image synthesis while providing rich control over scene parameters by learning the physical image formation process. While several supervised methods have been pro-posed for this task, acquiring a dataset of images with accurately aligned 3D models is very difficult. The main contribution of this work is to lift this restriction by training a neural rendering algorithm from unpaired data. We pro-pose an auto encoder for joint generation of realistic images from synthetic 3D models while simultaneously decomposing real images into their intrinsic shape and appearance properties. In contrast to a traditional graphics pipeline, our approach does not require to specify all scene properties, such as material parameters and lighting by hand.Instead, we learn photo-realistic deferred rendering from a small set of 3D models and a larger set of unaligned real images, both of which are easy to acquire in practice. Simultaneously, we obtain accurate intrinsic decompositions of real images while not requiring paired ground truth. Our experiments confirm that a joint treatment of rendering and de-composition is indeed beneficial and that our approach out-performs state-of-the-art image-to-image translation base-lines both qualitatively and quantitatively.

@inproceedings{Alhaija2020THREEDV,
  title = {Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition},
  author = {Alhaija, Hassan and Mustikovela, Siva and Jampani, Varun and Thies, Justus and Niessner, Matthias and Geiger, Andreas and Rother, Carsten},
  booktitle = {International Conference on 3D Vision (3DV)},
  year = {2020},
  doi = {}
}