Robust 3D Object Detection using Probabilistic Point Clouds from Single-Photon LiDARs

ICCV 2025 <br> Arxiv Project

teaser

Bhavya Goyal, Felipe Gutierrez-Barragan, Wei Lin, Andreas Velten, Yin Li, Mohit Gupta

University of Wisconsin-Madison

Abstract

LiDAR-based 3D sensors provide point clouds, a canonical 3D representation used in various scene understanding tasks. Modern LiDARs face key challenges in several real-world scenarios, such as long-distance or low-albedo objects, producing sparse or erroneous point clouds. These errors, which are rooted in the noisy raw LiDAR measurements, get propagated to downstream perception models, resulting in potentially severe loss of accuracy. This is because conventional 3D processing pipelines do not retain any uncertainty information from the raw measurements when constructing point clouds.

We propose Probabilistic Point Clouds (PPC), a novel 3D scene representation where each point is augmented with a probability attribute that encapsulates the measurement uncertainty (or confidence) in the raw data. We further introduce inference approaches that leverage PPC for robust 3D object detection; these methods are versatile and can be used as computationally lightweight drop-in modules in 3D inference pipelines. We demonstrate, via both simulations and real captures, that PPC-based 3D inference methods outperform several baselines using LiDAR as well as camera-LiDAR fusion models, across challenging indoor and outdoor scenarios involving small, distant, and low-albedo objects, as well as strong ambient light.

Installation

Follow the Installation steps for mmdetection3d framework. Or use my conda setup.

<details> <summary>My conda setup</summary>

conda create -n openmmlab python=3.8
conda activate openmmlab
pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu118
pip install mmcv==2.0.1 -f  https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/index.html
pip install mmdet==3.1.0
pip install -v -e .

</details>

Dataset

Download and extract the dataset (~12GB) file. Download the validation labels file. Use the following directory structure to organize the dataset.

.
.
└── data/          
|   └── sunrgbd/
|   |   └── sunrgbd_points_ppc/
|   |   |   └── sunrgbd_infos_val.pkl 
|   |   |   └── clean/
|   |   |   |   └── 0000001.bin
|   |   |   └── 1_100/
|   |   |   └── 1_50/
.
.
|   └── kitti/
|   |   └── kitti_points_ppc/
|   |   |   └── kitti_infos_val.pkl 
|   |   |   └── clean/
|   |   |   |   └── 0000000.bin
|   |   |   └── 5_1000/
.
.

If you need to evaluate on all SBR levels, you can download all sunrgbd_points_ppc_* files <a href="https://pages.cs.wisc.edu/~bhavya/ppcshared/data/sunrgbd/">here</a>.</summary>

<details> <summary>If you need to simulate PPCs yourself using different simulation parameters, or evaluate on a different dataset, you can use my simulation scripts.</summary>

Follow the original dataset instructions to prepare clean point cloud dataset.
Use ppc_simulate.sh to simulate 3D temporal waveforms. matlab is required.
```
cd tools/ppc_simulation
./ppc_simulate.sh 0 10
```
Use gen_points.sh to create probabilistic point clouds from the 3D waveforms.
```
./gen_points.sh 0 10
```
Use create_pkl.py to create label files for the whole dataset.
```
python create_pkl.py
```
Convert clean point clouds to ppc by adding probability 1 attribute.
```
python create_clean_ppc.py
```
Edit the dataset field in the scripts to simulate for KITTI dataset. Increase 10 to the size of the dataset to simulate all scenes.

</details>

Evaluation

Evaluate PPC model using ppc_test_votenet.sh script.

./ppc_test_votenet.sh 1_100 <model_weights.pth>

Use ppc_test_pvrcnn.sh and ppc_test_imvotenet.sh for PV-RCNN and ImVoteNet evaluation. Uncomment lines in the scripts for baseline evaluations.

Training

Train PPC model using ppc_train_votenet.sh script.

./ppc_train_votenet.sh

Uncomment lines in the scripts for baselines training.

Models

VoteNet

Evaluated on SUN RGBD validation dataset.

| Method | | | AP@25 | | | Download | |-------------------:|:--------:|:--------:|:---------:|:--------:|:--------:|:-------------------:| | | Clean | 0.1 | 0.05 | 0.02 | 0.01 | | | Matched Filtering | 51.34 | 42.43 | 38.77 | 16.95 | 11.34 | model | log| | Thresholding | 57.11 | 51.27 | 46.44 | 29.58 | 16.47 | model | log| | PPC | 58.61| 54.29| 52.46 |38.49 | 29.42| model | log|

PV-RCNN

Pedestrian mAP for PV-RCNN (3 class) model. Evaluated on KITTI val split using 11 recall positions for moderate difficulty.

| Method | | | mAP | | | Download | |-------------------:|:--------:|:--------:|:---------:|:--------:|:--------:|:-------------------:| | | Clean | 0.05 | 0.02 | 0.01 | 0.005 | | | Matched Filtering | 60.11 | 55.76 | 50.03 | 47.06 | 37.01 | model | log| | Thresholding | 61.62 | 57.72 | 54.80 | 49.23 | 38.62 | model | log| | PPC | 58.70 | 59.12| 59.04 |55.39 | 49.51| model | log|

ImVoteNet