PointRCNN

This is not the official implementation of PointRCNN. We add an image segmentation network to improve recall of point cloud segmentation. The 2-stage network is frustum pointNet. Any pull request is appreciated.

Introduction

A 3D object detector that takes point cloud and RGB image(optional) as input.

Results

Architecture

1. Perform foreground point segmentation on the whole point cloud
2. Output a 3D proposal box for every foreground point
3. Crop point cloud with proposal boxes and feed into the 2nd-stage classification and box refinement network

Usage

Dependencies

• python2.7
• tensorflow(1.10.0)
• shapely
• mayavi
• opencv-python
• Compile tensorflow operators for pointnet following to https://github.com/charlesq34/frustum-pointnets

Data Preparation

For trainning and validation, download KITTI 3d object detection dataset, and put the folders as

dataset/KITTI/object/
  training/
    calib/
    image_2/
    label_2/
    velodyne/

  testing/
    calib/
    image_2/
    velodyne/

For testing, download KITTI video sequence and calibration files, and put the folders as

2011_10_03/
  calib_cam_to_cam.txt
  calib_imu_to_velo.txt
  calib_velo_to_cam.txt
  image_02/
  velodyne_points/

[Optional] Scene Augmentation

Our implementation also supports using augmented scene point cloud for training RPN, please refer to the official implementation of PointRCNN. After generating the data, just put the aug_scene/ folder under dataset/KITTI/object. If you don't want to use it, just set use_aug_scene=False when using rpn_dataset.

Image segmentaion annotation

Our image segmentation network is DeelabV3+ official implementation. The senmantic segmentation annotation is obtained by the following steps:

• Pointcloud completion using ip_basic
• Project 3D points to image plane to get segmentation annotation Codes for finetuning can be founded at deeplab_kitti_object. Or you can just use your own image segmentation network.

Train

There are 3 sub-model to be trained.

Region Proposal Network

sh train_rpn.sh

Frustum Pointnet

Before training the 2-Stage Network, we need to save the output of RPN and Image Segmentaion Network to the disk first.

Image Segmentation Network

For now deeplabv3+ is used and finetune on KITTI 3D object dataset

Evaluate

Region Proposal Network

sh test_rpn.sh

This will save the output of RPN and Image segmentation network to ./rcnn_data_train for training the RCNN network.

Frustum Pointnet

test_frustum.sh

Test

End to end

sh test.sh

Evaluation

Point cloud segmentation

| Method | Coverage | Recall | Precision | | ---------- | -------- | ------ | --------- | | Point Only | 89.7% | 93.4% | 82.2% | | Point+Image| 93.5% | 97.0% | 76.6% |

Coverage means the percentage of object that have at least one point being detected.

Recall of RPN

Setting: IoU >= 0.5, 100 proposal

| Method | 3 Classes Recall | Car moderate | Pedastrian Moderate | Cyclist Moderate | | ---------- | ------------------- | ------------ | ------------------- | ---------------- | | Point+Image| 89% | 96% | 77% | 52% |

AP on Val Set

| Class | 3D mAP(Easy, Moderate, Hard) | BEV mAP(Easy, Moderate, Hard) | | ---------- | ----------------------------- |--------------------------------| | Car | 76.56 70.20 64.00 | 86.32 78.42 78.07 | | Pedestrain | 70.23 63.09 55.77 | 73.34 65.86 57.94 | | Cyclist | 76.89 50.91 50.28 | 78.27 59.00 51.63 |

Pretrained Models

| Model | Link | | ----- | ---- | | RPN |log_rpn.zip| | Image SegNet |log_rpn.zip| | Frustum PointNet |log_frustum.zip|

Reference

• Frustum PointNets for 3D Object Detection from RGB-D Data
• PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud