Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction (CVPR 2024)

Devikalyan Das, Christopher Wewer, Raza Yunus, Eddy Ilg, Jan Eric Lenssen

Saarland University, Saarland Informatics Campus, Germany and Max Planck Institute for Informatics, Saarland Informatics Campus, Germany

Project Page

Setup (with mamba)

1. Install dependencies

Use the below command

mamba env create --file env.yaml

Install the cuda-based gaussian rasterizer from here and simple_knn from here. These will be required for stage 2.

2. Download model weights

You can check the test images used in the paper and also download model weights from this link

3. Download data

You can download D-NeRF data from the here and UB4D data from here

4. Stage 1: Train on a monocular video

To train on a video for estimating the coarse 3D point clouds

python stage_1/run_train.py \
    -c <path to the config file> \
    -w <path to save the experiments/logs> \
    -d <path to the dataset directory> \
    -g 0 \
    -m train

5. Stage 1: Extract point cloud

To extract point clouds after training on a video or use the weights provided to extract the point cloud for DNeRF/UB4D

python stage_1/run_test.py \
    -c <path to the config file> \
    -w <path to save the experiments/logs> \
    -d <path to the dataset directory> \
    -g 0 \
    -m test \
    --test_iteration <iteration number>

6. Stage 2: Train Gaussian splatting

To train Gaussian splatting for skinning Gaussians on top of the estimated point cloud.

python stage_2/train.py \
    -s <path to the dataset directory> \
    -m <path to save the experiments/logs> \
    --npg_config <path to the config file for stage 2> \
    --npg_ckp <path to stage 1 checkpoint>

7. Stage 2: Render Views

To train Gaussian splatting for skinning Gaussians on top of the estimated point cloud.

python stage_2/render.py \
    -s <path to the dataset directory> \
    -m <path to save the experiments/logs> \
    --npg_config <path to the config file for stage 2> \
    --npg_ckp <path to stage 1 checkpoint>

Acknowledgement

Thanks a lot to all the authors for sharing their codes which played invaluable roles in our work!

• 3DGS
• Rasterizer

Citation

@inproceedings{das2024neural,
  title={Neural parametric gaussians for monocular non-rigid object reconstruction},
  author={Das, Devikalyan and Wewer, Christopher and Yunus, Raza and Ilg, Eddy and Lenssen, Jan Eric},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={10715--10725},
  year={2024}
}