Polynomial Neural Fields for Subband Decomposition and Manipulation

Pytorch implementation for the NeurIPS 2022 paper:

Polynomial Neural Fields for Subband Decomposition and Manipulation

Guandao Yang*, Sagie Benaim*, Varun Jampani, Kyle Genova, Jonathan T. Barron, Thomas Funkhouser, Bharath Hariharan, Serge Belongie (* Equal contribution.)

Introduction

Neural fields have emerged as a new paradigm for representing signals, thanks to their ability to do it compactly while being easy to optimize. In most applications, however, neural fields are treated like a black box, which precludes many signal manipulation tasks. In this paper, we propose a new class of neural fields called basis-encoded polynomial neural fields (PNFs). The key advantage of a PNF is that it can represent a signal as a composition of a number of manipulable and interpretable components without losing the merits of neural fields representation. We develop a general theoretical framework to analyze and design PNFs. We use this framework to design Fourier PNFs, which match state-of-the-art performance in signal representation tasks that use neural fields. In addition, we empirically demonstrate that Fourier PNFs enable signal manipulation applications such as texture transfer and scale-space interpolation.

Installation

This repository provides a Anaconda environment, and requires NVIDIA GPU to run the optimization routine. The environment can be set-up using the following commands:

conda env create -f environment.yml
conda activate PNF

Try Fitting PNF on Camera Men!

python train.py configs/camera_PNF_FF.yaml

<!-- TODO: Colab! - training, visualization of the laplacian pyramid ## Data Preparation ### TODO: DIV2K Data ### TODO: SDF Data (Armadillo) ### TODO: NeRF data prep (synthetic) ## Pretrained Models ### TODO: Pretrained model download scripts (img, sdf, and nerf) ## Training ### TODO: Bash script to train (DIV2K, SDF, and NeRF) ## Evaluation ### TODO: bash script to test with pretrained model on these datasets -->

Citation

If you find our paper or code useful, please cite us:

@inproceedings{yang2022pnf,
  title={Polynomial Neural Fields for Subband Decomposition and Manipulation},
  author={Yang, Guandao and Benaim, Sagie and Jampani, Varun and Genova, Kyle and Barron, Jonathan and Funkhouser, Thomas and Hariharan, Bharath and Belongie, Serge},
  booktitle={Thirty-Sixth Conference on Neural Information Processing Systems},
  year={2022}
}

Acknowledgement

This research was supported by the Pioneer Centre for AI, DNRF grant number P1. Guandao’s PhD was supported in part by research gifts from Google, Intel, and Magic Leap. Experiments are supported in part by Google clouds platform and GPUs donated by NVIDIA.