<div align='center'>

PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness

CVPR 2024 Oral, Best Paper Award Candidate

Anh-Quan Cao¹    Angela Dai²    Raoul de Charette¹   

<div> ¹ Inria ² Technical University of Munich </div> <br/>

</div>

If you find this work or code useful, please cite our paper and give this repo a star:

@InProceedings{cao2024pasco,
      title={PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness}, 
      author={Anh-Quan Cao and Angela Dai and Raoul de Charette},
      year={2024},
      booktitle = {CVPR}
}

Teaser

Table of Contents

• News
• 1. Installation
• 2. Data
  • 2.1. Semantic KITTI
  • 2.2. SSCBench-KITTI360
  • 2.3. Robo3D
• 3. Panoptic labels generation
  • 3.1. Semantic KITTI
  • 3.2. SSCBench-KITTI360
• 4. Training and evaluation
  • 4.1. Semantic KITTI
    • 4.1.1 Extract point features
    • 4.1.2 Training
    • 4.1.3 Evaluation
  • 4.2. SSCBench-KITTI360
    • 4.2.1 Training
    • 4.2.2 Evaluation
  • 4.3. Robo3D - SemanticKITTI-C
    • 4.3.1 Extract point features
    • 4.3.2 Evaluation
• 5. Visualization
• Acknowledgment

News

• 02/08/2024: Code for evaluation on Robo3D has been released.
• 30/07/2024: We added the training, evaluation, and checkpoints for PaSCo on SSCBench-KITTI360.
• 23/07/2024: The training/evaluation code and the checkpoint for PaSCo on SemanticKITTI has been released.
• 25/06/2024: Added visualization code.
• 10/06/2024: Training and evaluation code for PaSCo w/o MIMO has been released.
• 06/04/2024: Dataset download instructions and label generation code for SemanticKITTI are now available.
• 04/04/2024: PaSCo has been accepted as Oral paper at CVPR 2024 (0.8% = 90/11,532).
• 05/12/2023: Paper released on arXiv! Code will be released soon! Please watch this repo for updates.

1. Installation

1. Download the source code with git

   git clone https://github.com/astra-vision/PaSCo.git
   

2. Create conda environment:

   conda create -y -n pasco python=3.9
   conda activate pasco
   

3. Install pytorch 1.13.0

   pip install torch==1.13.0+cu117 torchvision==0.14.0+cu117 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/cu117
   

4. Install Minkowski Engine v0.5.4

5. Install pytorch_lightning 1.9.0 with torchmetrics 1.4.0.post0

   pip install --no-cache-dir pytorch_lightning==1.9.0
   

6. Install the additional dependencies:

   cd PaSCo/
   pip install -r requirements.txt
   

7. Install pytorch-scatter with torch 1.13.0 and CUDA 11.7:

   pip install torch-scatter -f https://data.pyg.org/whl/torch-1.13.0+cu117.html
   

8. Install PaSCo

   pip install -e ./
   

2. Data

2.1. Semantic KITTI

Please download the following data into a folder e.g. /gpfsdswork/dataset/SemanticKITTI and unzip:

• Semantic Scene Completion dataset v1.1 (SemanticKITTI voxel data (700 MB)) from SemanticKITTI website.

• Point-wise semantic labels (SemanticKITTI label data (179 MB)) from SemanticKITTI website.

• KITTI Odometry Benchmark calibration data (Download odometry data set (calibration files, 1 MB)).

• KITTI Odometry Benchmark Velodyne data (Download odometry data set (velodyne laser data, 80 GB)).

• The dataset folder at /gpfsdswork/dataset/SemanticKITTI should have the following structure:

  └── /gpfsdswork/dataset/SemanticKITTI
    └── dataset
      └── sequences
  

2.2. SSCBench-KITTI360

Please download the following data:

• The SSCBench-KITTI360 preprocess.sqf from SSCBench-KITTI360 official github and unsquash it into a folder e.g. /gpfsdswork/dataset/SSCBench-KITTI-360. I also uploaded the file preprocess.sqf here as backup.

• The KITTI-360 Raw Velodyne Scans (119G) from KITTI-360 download page and put into folder e.g. /gpfsdswork/dataset/KITTI-360.

• The folder /gpfsdswork/dataset/KITTI-360 should have the following structure:

  /gpfsdswork/dataset/KITTI-360
  └── data_3d_raw
      ├── 2013_05_28_drive_0000_sync
      ├── 2013_05_28_drive_0002_sync
      ├── 2013_05_28_drive_0003_sync
      ├── 2013_05_28_drive_0004_sync
      ├── 2013_05_28_drive_0005_sync
      ├── 2013_05_28_drive_0006_sync
      ├── 2013_05_28_drive_0007_sync
      └── 2013_05_28_drive_0009_sync
  

• The folder /gpfsdswork/dataset/SSCBench-KITTI-360 should have the following structure:

  /gpfsdswork/dataset/SSCBench-KITTI-360/
  ├── labels                        
  │   ├── 2013_05_28_drive_0000_sync
  │   ├── 2013_05_28_drive_0002_sync
  │   ├── 2013_05_28_drive_0003_sync
  │   ├── 2013_05_28_drive_0004_sync
  │   ├── 2013_05_28_drive_0005_sync
  │   ├── 2013_05_28_drive_0006_sync
  │   ├── 2013_05_28_drive_0007_sync
  │   ├── 2013_05_28_drive_0009_sync
  │   └── 2013_05_28_drive_0010_sync
  ├── labels_half                   
  │   ├── 2013_05_28_drive_0000_sync
  │   ├── 2013_05_28_drive_0002_sync
  │   ├── 2013_05_28_drive_0003_sync
  │   ├── 2013_05_28_drive_0004_sync
  │   ├── 2013_05_28_drive_0005_sync
  │   ├── 2013_05_28_drive_0006_sync
  │   ├── 2013_05_28_drive_0007_sync  
  │   ├── 2013_05_28_drive_0009_sync
  │   └── 2013_05_28_drive_0010_sync
  ├── README            
  └── unified                         
  └── labels
  

2.3. Robo3D

1. Please download the SemanticKITTI-C following the official instruction and put in a folder e.g. /gpfsdswork/dataset/SemanticKITTI-C.
2. The folder /gpfsdswork/dataset/SemanticKITTI-C should have the following structure:

   /gpfsdswork/dataset/SemanticKITTI-C
   ├── beam_missing
   ├── cross_sensor
   ├── crosstalk
   ├── fog
   ├── incomplete_echo
   ├── motion_blur
   ├── snow
   └── wet_ground
   

3. Panoptic labels generation

3.1. Semantic KITTI

1. Create a folder to store preprocess data for Semantic KITTI dataset e.g. /lustre/fsn1/projects/rech/kvd/uyl37fq/pasco_preprocess/kitti .
2. Execute the command below to generate panoptic labels, or move to the next step to directly download the pre-generated labels:

   cd PaSCo/
   python label_gen/gen_instance_labels.py \
       --kitti_config=pasco/data/semantic_kitti/semantic-kitti.yaml \
       --kitti_root=/gpfsdswork/dataset/SemanticKITTI \
       --kitti_preprocess_root=/lustre/fsn1/projects/rech/kvd/uyl37fq/pasco_preprocess/kitti \
       --n_process=10
   

[!NOTE] This command processes 4649 files on CPU took approximately 10 hours using 10 processes. The number of processes can be adjusted by modifying the n_process parameter.

3. You can download the generated panoptic labels for Semantic KITTI:
   1. Go to the preprocess folder for KITTI:

      cd /lustre/fsn1/projects/rech/kvd/uyl37fq/pasco_preprocess/kitti
      

   2. Download the compressed file:

      wget https://github.com/astra-vision/PaSCo/releases/download/v0.0.1/kitti_instance_label_v2.tar.gz
      

   3. Extract the file:

      tar xvf kitti_instance_label_v2.tar.gz
      

4. Your folder structure should look as follows:

   /lustre/fsn1/projects/rech/kvd/uyl37fq/pasco_preprocess/kitti
   └── instance_labels_v2
       ├── 00
       ├── 01
       ├── 02
       ├── 03
       ├── 04
       ├── 05
       ├── 06
       ├── 07
       ├── 08
       ├── 09
       └── 10
   

3.2. SSCBench-KITTI360

1. Create a folder to store preprocess data for SSCBench-KITTI360 dataset e.g. /lustre/fsn1/projects/rech/kvd/uyl37fq/pasco_preprocess/kitti360 .
2. Execute the command below to generate panoptic labels (took approximately 8.5 hours to process 12464 scans using 10 processes), or move to the next step to directly download the pre-generated labels:

   cd PaSCo/
   python label_gen/gen_instance_labels_kitti360.py \
       --kitti360_label_root=/gpfsdswork/dataset/SSCBench-KITTI-360 \
       --kitti360_preprocess_root=/lustre/fsn1/projects/rech/kvd/uyl37fq/pasco_preprocess/kitti360 \
       --n_process=10
   

3. You can download the generated panoptic labels for SSCBench-KITTI360:
   1. Go to the preprocess