KeypointNeRF
KeypointNeRF Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints
Install / Use
/learn @facebookresearch/KeypointNeRFREADME
News :new:
- [2022/10/01] Combine ICON with our relative spatial keypoint encoding for fast and convenient monocular reconstruction, without requiring the expensive SMPL feature. More details are here.
Installation
Please install python dependencies specified in environment.yml:
conda env create -f environment.yml
conda activate KeypointNeRF
Data preparation
Please see DATA_PREP.md to setup the ZJU-MoCap dataset.
After this step the data directory follows the structure:
./data/zju_mocap
├── CoreView_313
├── CoreView_315
├── CoreView_377
├── CoreView_386
├── CoreView_387
├── CoreView_390
├── CoreView_392
├── CoreView_393
├── CoreView_394
└── CoreView_396
Train your own model on the ZJU dataset
Execute train.py script to train the model on the ZJU dataset.
python train.py --config ./configs/zju.json --data_root ./data/zju_mocap
After the training, the model checkpoint will be stored under ./EXPERIMENTS/zju/ckpts/last.ckpt, which is equivalent to the one provided here.
Evaluation
To extract render and evaluate images, execute:
python train.py --config ./configs/zju.json --data_root ./data/zju_mocap --run_val
python eval_zju.py --src_dir ./EXPERIMENTS/zju/images_v3
To visualize the dynamic results, execute:
python render_dynamic.py --config ./configs/zju.json --data_root ./data/zju_mocap --model_ckpt ./EXPERIMENTS/zju/ckpts/last.ckpt
We compare KeypointNeRF with recent state-of-the-art methods. The evaluation metric is SSIM and PSNR. | Models | PSNR ↑ | SSIM ↑ | |---|---|---| | pixelNeRF <font size="1">(Yu et al., CVPR'21)</font> | 23.17 | 86.93 | | PVA <font size="1">(Raj et al., CVPR'21)</font> | 23.15 | 86.63 | | NHP <font size="1">(Kwon et al., NeurIPS'21)</font> | 24.75 | 90.58 | | KeypointNeRF* <font size="1">(Mihajlovic et al., ECCV'22)</font> | 25.86 | 91.07 |
<p align="left" style="font-size:10px"> (*Note that results of KeypointNeRF are slightly higher compared to the numbers reported in the original paper due to training views not beeing shuffled during training.) </p>Reconstruction from a Single Image
Our relative spatial encoding can be used to reconstruct humans from a single image. As a example, we leverage ICON and replace its expensive SDF feature with our relative spatial encoding.
<div align="center"><img src="./assets/icon_vs_kpts.png" alt="Logo" width="100%"> While it achieves comparable quality to ICON, it's much <strong>faster</strong> and more <strong>convinient</strong> to use <span align="left" style="font-size:8px">(*displayed image taken from pinterest.com)</span>. </div>3D Human Reconstruction on CAPE
| Models | Chamfer ↓ (cm) | P2S ↓ (cm) | |---|---|---| | PIFu <font size="1">(Saito et al., ICCV'19)</font> | 3.573 | 1.483 | | ICON <font size="1">(Xiu et al., CVPR'22)</font> | 1.424 | 1.351 | | KeypointICON <font size="1">(Mihajlovic et al., ECCV'22; Xiu et al., CVPR'22)</font> | 1.539 | 1.358 |
Check the benchmark here and more details here.
Publication
If you find our code or paper useful, please consider citing:
@inproceedings{Mihajlovic:ECCV2022,
title = {{KeypointNeRF}: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints},
author = {Mihajlovic, Marko and Bansal, Aayush and Zollhoefer, Michael and Tang, Siyu and Saito, Shunsuke},
booktitle={European conference on computer vision},
year={2022},
}
License
CC-BY-NC 4.0. See the LICENSE file.
