Gen3D

[CSUR 2023] A Survey on Deep Generative 3D-aware Image Synthesis

Generate Convert Improve

Install / Use

/learn @weihaox/Gen3D

About this skill

Quality Score

0/100

README

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"> <head> <meta name="generator" content="jemdoc, see http://jemdoc.jaboc.net/" /> <meta http-equiv="Content-Type" content="text/html;charset=utf-8" /> <link rel="stylesheet" href="jemdoc.css" type="text/css" /> </head>  <div id="layout-content">  <h1 align="center">A Survey on Deep Generative 3D-aware Image Synthesis</h1> ACM Computing Surveys, 2023 <a href="https://weihaox.github.io/">Weihao Xia</a> · <a href="http://www.homepages.ucl.ac.uk/~ucakjxu/">Jing-Hao Xue</a> <a href='https://arxiv.org/abs/2210.14267'> <img src='https://img.shields.io/badge/Paper-Paper-green?style=flat&logo=arxiv&logoColor=green' alt='arxiv Paper'> </a> <a href='https://weihaox.github.io/Gen3D/' style='padding-left: 0.5rem;'> <img src='https://img.shields.io/badge/Project-Page-blue?style=flat&logo=Google%20chrome&logoColor=blue' alt='Project Page'> </a> <a href='https://dl.acm.org/doi/10.1145/3626193' style='padding-left: 0.5rem;'> <img src='https://img.shields.io/badge/CSUR-Paper-red?style=flat&logoColor=red' alt='CSUR Paper'> </a>

Introduction

This project lists representative papers/codes/datasets about deep 3D-aware image synthesis. Besides 3D-aware Generative Models (GANs and Diffusion Models) discussed in this survey, this project additionally covers novel view synthesis studies, especially those based on implicit neural representations such as NeRF.

We aim to constantly update the latest relevant papers and help the community track this topic. Please feel free to join us and contribute to the project. Please do not hesitate to reach out if you have any questions or suggestions.

Survey paper

A Survey on Deep Generative 3D-aware Image Synthesis
Weihao Xia and Jing-Hao Xue. ACM Computing Surveys, 2023.

3D Control of 2D GANs

3D Control Latent Directions

For 3D control over diffusion models simiar to GAN, please refer to diffusion latent editing.

SeFa: Closed-Form Factorization of Latent Semantics in GANs. Yujun Shen, Bolei Zhou. CVPR 2021. [Paper] [Project] [Code]
GANSpace: Discovering Interpretable GAN Controls. Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, Sylvain Paris. NeurIPS 2020. [Paper] [Code]
Interpreting the Latent Space of GANs for Semantic Face Editing. Yujun Shen, Jinjin Gu, Xiaoou Tang, Bolei Zhou. CVPR 2020. [Paper] [Project] [Code]
Unsupervised Discovery of Interpretable Directions in the GAN Latent Space. Andrey Voynov, Artem Babenko. ICML 2020. [Paper] [Code]
On the "steerability" of generative adversarial networks. Ali Jahanian, Lucy Chai, Phillip Isola. ICLR 2020. [Paper] [Project] [Code]

3D Parameters as Controls

3D-FM GAN: Towards 3D-Controllable Face Manipulation. Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, and Sun-Yuan Kung. ECCV 2022. [Paper] [Project]
GAN-Control: Explicitly Controllable GANs. Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, Gerard Medioni. ICCV 2021. [Paper] [Project] [Code]
CONFIG: Controllable Neural Face Image Generation. Marek Kowalski, Stephan J. Garbin, Virginia Estellers, Tadas Baltrušaitis, Matthew Johnson, Jamie Shotton. ECCV 2020. [Paper] [Code]
DiscoFaceGAN: Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning. Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, Xin Tong. CVPR 2020. [Paper] [Code]
StyleRig: Rigging StyleGAN for 3D Control over Portrait Images. Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt. CVPR 2020 (oral). [Paper] [Project]
PIE: Portrait Image Embedding for Semantic Control. Ayush Tewari, Mohamed Elgharib, Mallikarjun B R., Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt. TOG (SIGGRAPH Asia) 2020. [Paper] [Project]

3D Prior Knowledge as Constraints

3D-Aware Indoor Scene Synthesis with Depth Priors. Zifan Shi, Yujun Shen, Jiapeng Zhu, Dit-Yan Yeung, Qifeng Chen. ECCV 2022 (oral). [Paper] [Project] [Code]
NGP: Towards a Neural Graphics Pipeline for Controllable Image Generation. Xuelin Chen, Daniel Cohen-Or, Baoquan Chen, Niloy J. Mitra. Eurographics 2021. [Paper] [Code]
Lifting 2D StyleGAN for 3D-Aware Face Generation. Yichun Shi, Divyansh Aggarwal, Anil K. Jain. CVPR 2021. [Paper] [Code]
RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis. Atsuhiro Noguchi, Tatsuya Harada. ICLR 2020. [Paper] [Code]
Visual Object Networks: Image Generation with Disentangled 3D Representation. Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, William T. Freeman. NeurIPS 2018. [Paper] [Project] [Code]
3D Shape Induction from 2D Views of Multiple Objects. Matheus Gadelha, Subhransu Maji, Rui Wang. 3DV 2017. [Paper] [Project] [Code]
Generative Image Modeling using Style and Structure Adversarial Networks. Xiaolong Wang, Abhinav Gupta. ECCV 2016. [Paper] [Project] [Code]

3D-aware GANs for a Single Image Category

Unconditional 3D Generative Models

BallGAN: 3D-aware Image Synthesis with a Spherical Background. Minjung Shin, Yunji Seo, Jeongmin Bae, Young Sun Choi, Hyunsu Kim, Hyeran Byun, Youngjung Uh. ICCV 2023. [Paper] [Project] [Code]
Mimic3D: Thriving 3D-Aware GANs via 3D-to-2D Imitation. Xingyu Chen, Yu Deng, Baoyuan Wang. ICCV 2023. [Paper] [Project]
GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds. Jianfeng Xiang, Jiaolong Yang, Yu Deng, Xin Tong. ICCV 2023. [Paper] [Project]
Live 3D Portrait: Real-Time Radiance Fields for Single-Image Portrait View Synthesis. Alex Trevithick, Matthew Chan, Michael Stengel, Eric R. Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, Koki Nagano. TOG (SIGGRAPH) 2023. [Paper] [Project]
VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids. Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, Andreas Geiger. NeurIPS 2022. [Paper] [Code]
GeoD: Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator. Zifan Shi, Yinghao Xu, Yujun Shen, Deli Zhao, Qifeng Chen, Dit-Yan Yeung. NeurIPS 2022. [Paper] [[Project

Related Skills

node-connect

344.1k

Diagnose OpenClaw node connection and pairing failures for Android, iOS, and macOS companion apps

frontend-design

96.8k

Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.

openai-whisper-api

344.1k

Transcribe audio via OpenAI Audio Transcriptions API (Whisper).

qqbot-media

344.1k

QQBot 富媒体收发能力。使用 <qqmedia> 标签，系统根据文件扩展名自动识别类型（图片/语音/视频/文件）。