SPARF: Neural Radiance Fields from Sparse and Noisy Poses

This is not an officially supported Google product.

This repository contains the code for the paper: SPARF: Neural Radiance Fields from Sparse and Noisy Poses. In CVPR, 2023 (Highlight).

Authors: Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari

[arXiv preprint] [Website] [Youtube teaser]

Our approach SPARF produces realistic novel-view rendering given as few as 2 or 3 input images, with noisy camera poses. We add two novel constraints into the pose-NeRF optimization: the multi-view correspondence loss and the depth-consistency loss.

Please contact Prune Truong (prune.truong@vision.ee.ethz.ch) if you have any questions!

We provide PyTorch code for all experiments: BARF/SPARF for joint pose-NeRF training, NeRF/SPARF when considering fixed ground-truth poses as input.

Citation

If you find our code useful for your research, please cite

@inproceedings{sparf2023,
  title={SPARF: Neural Radiance Fields from Sparse and Noisy Poses},
  author = {Truong, Prune and Rakotosaona, Marie-Julie and Manhardt, Fabian and Tombari, Federico},
  publisher = {{IEEE/CVF} Conference on Computer Vision and Pattern Recognition, {CVPR}},
  year = {2023}
}

Installation

This code is developed with Python3 (python3) with Cuda 11.3. All models were trained on single A100 (or V100) GPU with 20GB of memory. This code does NOT support multi-GPU training. Please use the following command for installation.

It is recommended to use Anaconda to set up the environment. Install the dependencies and activate the environment sparf-env with

conda create -n sparf-env python=3
conda activate sparf-env
pip install -r requirements.txt

For the specific versions of the packages used to develop this model, run the following instead.

pip install -r requirements_w_version.txt

Make sure to the dependencies of the submodules are downloaded. Everything should be included in the provided requirement file, except for cupy: We use PDC-Net to extract correspondences between the views, for which CuPy is a required dependency. It can be installed using pip install cupy or alternatively using one of the provided binary packages as outlined in the CuPy repository. For another CUDA version, change accordingly.

pip install cupy-cuda113 --no-cache-dir

Initialize the external submodule dependencies with

git submodule update --init --recursive
git submodule update --recursive --remote

Optional: The submodule contains lots of unnecessary files. You can suppress them by running

bash third_party/remove_unused_files.sh

Create source/admin/local.py by running the following command and update the paths to the datasets and workdirs. We provide an example admin/local_example.py where all datasets are stored in data/.

python -c "from source.admin.environment import create_default_local_file; create_default_local_file()"

Download the pre-trained model of PDC-Net here. You will need to change the path to the pre-trained model in train_settings/default_config.py cfg.flow_ckpt_path (L.190).

<details> <summary>Make sure the correspondence network is installed and running correctly</summary>

Run the following command

python third_party/test_pdcnet_installation.py

You should obtain the following image as third_party/test_pdcnet.png

</details>

Datasets

Images: We use the DTU dataset, processed by PixelNeRF, where the images are processed and resized to 300 x 400. Download the data here.
Mask Data: For evaluation, we report also masked metrics. For this, we use the object masks provided by DVR, IDR and RegNeRF. RegNeRF provides the full mask data (for the test splits), which you can download here.
Ground-truth depth maps: For evaluation, we report the depth error. For this, we download the depth maps. They are extracted from MVSNeRF.

</details> <details> <summary>LLFF</summary>

The LLFF real-world data can be found in the NeRF Google Drive. For convenience, you can download them with the following script: (under this repo)

gdown --id 16VnMcF1KJYxN9QId6TClMsZRahHNMW5g # download nerf_llff_data.zip
unzip nerf_llff_data.zip
rm -f nerf_llff_data.zip
mv nerf_llff_data data/llff

</details> <details> <summary>Replica</summary>

You can download the replica dataset with the following script:

# you can also download the Replica.zip manually through
# link: https://caiyun.139.com/m/i?1A5Ch5C3abNiL password: v3fY (the zip is split into smaller zips because of the size limitation of caiyun)
wget https://cvg-data.inf.ethz.ch/nice-slam/data/Replica.zip
unzip Replica.zip

</details>

I. Running the code

Training: Quick start

The installation should have generated a local configuration file "admin/local.py". In case the file was not generated, run python -c "from source.admin.environment import create_default_local_file; create_default_local_file()"to generate it.

Next, set the paths to the training workspace self.workspace_dir, i.e. the directory where the model weights and checkpoints will be saved.
Set the path to self.tensorboard_dir, where the tensorboard files will be stored
Set the path to self.log_dir, where the logs will be stored
Set the path to self.eval_dir, where the test results will be stored.
Also set the paths to the datasets you want to use (and which should be downloaded beforehand, see above).
Set the path to the PDC-Net pre-trained model (previously downloaded) in train_settings/default_config.py cfg.flow_ckpt_path.

If all the dependencies have been correctly installed, you can train/evaluate a network using the run_trainval.py script in the correct conda environment. All checkpointing, logging and saving metrics should be done automatically.

conda activate sparf-env
# Selecting <train_module> <train_name> <nbr_input_views> <scene_name>
python run_trainval.py <train_module> <train_name> --train_sub <nbr_input_views> --scene <scene_name>

Here, train_module is the sub-module inside train_settings and train_name is the name of the train setting file to be used.

The snapshots along with a video of the RGB and depth renderings will be saved in the directory workspace_dir/<train_module>/subset_<nbr_input_views>/<scene_name>/<train_name>.

The corresponding tensorboard file will be stored in tensorboard_dir/<train_module>/subset_<nbr_input_views>/<scene_name>/<train_name>.

where workspace_dir and tensorboard_dir were set in the source.admin.local.py file.

Running this command directly computes results on the test set at the end of training, which will be stored in eval_dir/dataset_name/<train_module>/subset_<nbr_input_views>/<scene_name>/<train_name>. If you wish to recompute them, or re-generate the video rendering, run:

# Selecting <train_module> <train_name> <nbr_input_views> <scene_name>
# to regenerate the test metrics 
python run_trainval.py <train_module> <train_name> --train_sub <nbr_input_views> --scene <scene_name> --test_metrics_only True 

# to regenerate the video of the renderings
python run_trainval.py <train_module> <train_name> --train_sub <nbr_input_views> --scene <scene_name> --render_video_only True

Training: Example

The configs are found in train_settings

For example, you can train using the included default SPARF settings for joint pose-nerf training on DTU, starting from initial noisy poses (3 views) by running:

python run_trainval.py joint_pose_nerf_training/dtu sparf --train_sub 3 --scene scan82

All the snapshots will be stored in the directory workspace_dir/joint_pose_nerf_training/dtu/subset_3/scan82/sparf. All the tensorboard will be stored in tensorboard_dir/joint_pose_nerf_training/dtu/subset_3/scan82/sparf.

Running this command directly computes results on the test set at the end of training, which will be stored in eval_dir/dtu/joint_pose_nerf_training/dtu/subset_3/scan82/sparf.

Visualizing the results

We have included code to visualize the training over TensorBoard. The TensorBoard events include the following:

SCALARS: the rendering losses and PSNR over the course of optimization. For joint pose-NeRF optimization, the rotational/translational errors with respect to the given poses are also computed. The validation metrics are als

Sparf

Install / Use

README