RoGs: Large Scale Road Surface Reconstruction with Meshgrid Gaussian

<p align="center"> <!-- project badges --> <a href="https://github.com/fzhiheng/RoGS.git"><img src="https://img.shields.io/badge/Project-Page-ffa"/></a> <!-- paper badges --> <a href="https://arxiv.org/abs/2405.14342"> <img src='https://img.shields.io/badge/arXiv-Page-aff'> </a> </p> <p align="center"> <img src="docs/image/kitti.png" width="90%"/> </p>

Setup

Tested with Pytorch 1.13.1 and CUDA 11.7 and Pytoch3d 0.7.8

Clone the repo.

git clone https://github.com/fzhiheng/RoGS.git

Environment setup

1. conda create -n rogs python=3.8.10 -y
   conda activate rogs
   pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
   pip install addict PyYAML tqdm scipy pytz plyfile opencv-python pyrotation pyquaternion nuscenes-devkit
   

2. Install pytorch3d.

3. Install the diff-gaussian-rasterization with orthographic camera

   git clone --recursive https://github.com/fzhiheng/diff-gs-depth-alpha.git && cd diff-gs-depth-alpha
   python setup.py install
   cd ..
   

4. Install the diff-gaussian-rasterization to optimize semantic

   In order to optimize the semantic and try not to lose performance (there is some performance loss in dynamically allocating memory when the channel does not need to be specified). We still use the library above. Only a few changes are needed.

   git clone --recursive https://github.com/fzhiheng/diff-gs-depth-alpha.git diff-gs-label && cd diff-gs-label
   mv diff_gaussian_rasterization diff_gs_label
   
   # follow the instructions below to modify the file
   
   python setup.py install
   

   Set NUM_CHANNELS in file cuda_rasterizer/config.h to num_class ( 7 for nuScenes and 5 for KITTI) and change all diff_gaussian_rasterization in setup.py to diff_gs_label. On the dataset KITTI, we changed the name of the library to diff_gs_label2 . In practice, you can set NUM_CHANNELS according to the category of your semantic segmentation and change the name of the library.

Dataset

nuScenes

In configs/local_nusc.yaml and configs/local_nusc_mini.yaml

• base_dir: Put official nuScenes here, e.g. {base_dir}/v1.0-trainval

• image_dir: Put segmentation results here. We use the segmentation results provided by Rome. You can download here.

• road_gt_dir：Put ground truth here. To produce ground truth:

  python -m preprocess.process --nusc_root /dataset/nuScenes/v1.0-mini --seg_root /dataset/nuScenes/nuScenes_clip ---save_root /dataset/nuScenes/ -v mini --scene_names scene-0655
  

KITTI

In configs/local_kitti.yaml

• base_dir: Put official kitti odometry dataset here, e.g. {base_dir}/sequences

• image_dir: Put segmentation results here. We use the segmentation results provided by Rome. You can download here.

• pose_dir: Put kitti odometry pose here, e.g. {pose_dir}/dataset/poses

Optimization

python train.py --config configs/local_nusc_mini.yaml

Citation

@article{feng2024rogs,
  title={RoGs: Large Scale Road Surface Reconstruction with Meshgrid Gaussian},
  author={Feng, Zhiheng and Wu, Wenhua and Deng, Tianchen and Wang, Hesheng},
  journal={arXiv preprint arXiv:2405.14342},
  year={2024}
}

Acknowledgements

This project largely references 3D Gaussian Splatting and RoMe. Thanks for their amazing works!