Intelligent Systems
Note: This research group has relocated. Discover the updated page here

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

2020

Conference Paper

avg


While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera and scene properties, eg, the object identity may vary with the viewpoint. In this paper, we propose a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene. In contrast to voxel-based representations, radiance fields are not confined to a coarse discretization of the 3D space, yet allow for disentangling camera and scene properties while degrading gracefully in the presence of reconstruction ambiguity. By introducing a multi-scale patch-based discriminator, we demonstrate synthesis of high-resolution images while training our model from unposed 2D images alone. We systematically analyze our approach on several challenging synthetic and real-world datasets. Our experiments reveal that radiance fields are a powerful representation for generative image synthesis, leading to 3D consistent models that render with high fidelity.

Author(s): Katja Schwarz and Yiyi Liao and Michael Niemeyer and Andreas Geiger
Book Title: Advances in Neural Information Processing Systems 33
Volume: 25
Pages: 20154--20166
Year: 2020
Month: December
Editors: Larochelle , H. and Ranzato, M. and Hadsell , R. and Balcan , M. F. and Lin, H.
Publisher: Curran Associates, Inc.

Department(s): Autonomous Vision
Bibtex Type: Conference Paper (inproceedings)
Paper Type: Conference

Event Name: 34th Conference on Neural Information Processing Systems (NeurIPS 2020)
Event Place: Online

Address: Red Hook, NY
ISBN: 978-1-7138-2954-6
State: Published
URL: https://papers.nips.cc/paper_files/paper/2020/hash/e92e1b476bb5262d793fd40931e0ed53-Abstract.html

Links: pdf
suppmat
video
Project Page
Video:

BibTex

@inproceedings{Schwarz2020NEURIPS,
  title = {GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis},
  author = {Schwarz, Katja and Liao, Yiyi and Niemeyer, Michael and Geiger, Andreas},
  booktitle = {Advances in Neural Information Processing Systems 33},
  volume = {25},
  pages = {20154--20166},
  editors = {Larochelle , H. and Ranzato, M. and Hadsell , R. and Balcan , M. F. and Lin, H.},
  publisher = {Curran Associates, Inc.},
  address = {Red Hook, NY},
  month = dec,
  year = {2020},
  doi = {},
  url = {https://papers.nips.cc/paper_files/paper/2020/hash/e92e1b476bb5262d793fd40931e0ed53-Abstract.html},
  month_numeric = {12}
}