VMAP

<!-- PROJECT LOGO --> <p align="center"> <h1 align="center">vMAP: Vectorised Object Mapping for Neural Field SLAM</h1> <p align="center"> <a href="https://kxhit.github.io"><strong>Xin Kong</strong></a> · <a href="https://shikun.io"><strong>Shikun Liu</strong></a> · <a href="https://marwan99.github.io/"><strong>Marwan Taher</strong></a> · <a href="https://www.doc.ic.ac.uk/~ajd/"><strong>Andrew Davison</strong></a> </p> <h3 align="center"><a href="https://arxiv.org/abs/2302.01838">Paper</a> | <a href="https://youtu.be/_H_jNnhUAsE">Video</a> | <a href="https://kxhit.github.io/vMAP">Project Page</a></h3> <div align="center"></div> <p align="center"> <a href=""> <img src="./media/teaser.png" alt="Logo" width="80%"> </a> </p> <p align="center"> vMAP builds an object-level map from a real-time RGB-D input stream. Each object is represented by a separate MLP neural field model, all optimised in parallel via vectorised training. </p> <br>

We provide the implementation of the following neural-field SLAM frameworks:

• vMAP [Official Implementation]
• iMAP [Simplified and Improved Re-Implementation, with depth guided sampling]

Install

First, let's start with a virtual environment with the required dependencies.

conda env create -f environment.yml

Dataset

Please download the following datasets to reproduce our results.

• Replica Demo - Replica Room 0 only for faster experimentation.
• Replica - All Pre-generated Replica sequences. For Replica data generation, please refer to directory data_generation.
• ScanNet - Official ScanNet sequences. Each dataset contains a sequence of RGB-D images, as well as their corresponding camera poses, and object instance labels. To extract data from ScanNet .sens files, run

  conda activate py2
  python2 reader.py --filename ~/data/ScanNet/scannet/scans/scene0024_00/scene0024_00.sens --output_path ~/data/ScanNet/objnerf/ --export_depth_images --export_color_images --export_poses --export_intrinsics
  

Config

Then update the config files in configs/.json with your dataset paths, as well as other training hyper-parameters.

"dataset": {
        "path": "path/to/ims/folder/",
    }

Running vMAP / iMAP

The following commands will run vMAP / iMAP in a single-thread setting.

vMAP

python ./train.py --config ./configs/Replica/config_replica_room0_vMAP.json --logdir ./logs/vMAP/room0 --save_ckpt True

iMAP

python ./train.py --config ./configs/Replica/config_replica_room0_iMAP.json --logdir ./logs/iMAP/room0 --save_ckpt True

Evaluation

To evaluate the quality of reconstructed scenes, we provide two different methods,

3D Scene-level Evaluation

The same metrics following the original iMAP, to compare with GT scene meshes by Accuracy, Completion and Completion Ratio.

python ./metric/eval_3D_scene.py

3D Object-level Evaluation

We also provide the object-level metrics by computing the same metrics but averaging across all objects in a scene.

python ./metric/eval_3D_obj.py

Results

We provide raw results, including 3D meshes, 2D novel view rendering, and evaluated metrics of vMAP and iMAP* for easier comparison.

• Replica

Acknowledgement

We would like thank the following open-source repositories that we have build upon for the implementation of this work: NICE-SLAM, and functorch.

Citation

If you found this code/work to be useful in your own research, please considering citing the following:

@article{kong2023vmap,
  title={vMAP: Vectorised Object Mapping for Neural Field SLAM},
  author={Kong, Xin and Liu, Shikun and Taher, Marwan and Davison, Andrew J},
  journal={arXiv preprint arXiv:2302.01838},
  year={2023}
}

@inproceedings{sucar2021imap,
  title={iMAP: Implicit mapping and positioning in real-time},
  author={Sucar, Edgar and Liu, Shikun and Ortiz, Joseph and Davison, Andrew J},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={6229--6238},
  year={2021}
}