3D-LiDAR and camera extrinsic calibration [paper][arxiv]

Citation

If you find our codes and method useful to your work, please consider citing this work:

@Article{WANG2017Lidar_camera_cali,
AUTHOR = {Wang, Weimin and Sakurada, Ken and Kawaguchi, Nobuo},
TITLE = {Reflectance Intensity Assisted Automatic and Accurate Extrinsic Calibration of 3D LiDAR and Panoramic Camera Using a Printed Chessboard},
JOURNAL = {Remote Sensing},
VOLUME = {9},
YEAR = {2017},
NUMBER = {8},
ARTICLE-NUMBER = {851},
ISSN = {2072-4292},
DOI = {10.3390/rs9080851}
}

Introduction

This is an python implementation for the fully automatic and accurate extrinsic calibration of an 3D-LiDAR and the camera based on the laser's reflectance intensity. <br> The paper is available here.<br> The main features of this implementations are:<br>

automatic segmentation of the point cloud acquired by Velodyne 3D LiDAR
automatic detection of the chessboard
automatic corners detection from the chessboard's point cloud
optimization for extrinsic calibration parameters
various of visualization for 3D point clouds with VTK python wrapper<br> These features are implemented for VLP-16, HDL-32e and HDL-64e. However, they tested only with HDL-32e. We are appreciated if one could provide data of other types for testing. <img src="readme_files/VLP16_omni60.gif" />

Updates

2018-06-05 Add instruction for installing dependencies on Ubuntu.
2018-04-16 (Release of Version 0.2)
- Implement calibration for monocular camera.
- Add sample data and results for perspective camera calibration.
- Add a feature that can hide occluded parts by the chessboard when project the point cloud to the image.
- Some other minor changes.

Dependencies (Tested on macOS sierra and Ubuntu 14.04/16.04)

Python >= 2.7.9

OpenCV (Remember to enable python shared library if you are using python installed by pyenv)

for macOS:<br>

brew install opencv3
echo /usr/local/opt/opencv3/lib/python2.7/site-packages >> /usr/local/lib/python2.7/site-packages/opencv3.pth

for Ubuntu:<br>

git clone https://github.com/opencv/opencv.git
cd opencv && mkdir build && cd build && cmake .. && make -j4 && make install

OpenGV

for macOS and Ubuntu:<br>

git clone https://github.com/mfxox/opengv
cd opengv
mkdir build && cd build && cmake .. && make && make install

Point Cloud Library (PCL)
- for macOS:<br>
```
brew install pcl
```
- for Ubuntu:<br>
```
sudo apt-get install libpcl-1.7-all
```
or build from source

PCL python bindings

for macOS and Ubuntu:<br>

git clone https://github.com/mfxox/python-pcl
cd python-pcl
python setup.py install

MATLAB engine for Python:
- for macOS and Linux:<br>
```
cd "matlabroot/extern/engines/python"
python setup.py install
```
- MATLAB python is used for corner detection from panoramic images. The OpenCV backend is also available which can be set by the backend parameter in config.yaml, however, Opencv may fail to detect the corners. You can also use the example files (output/img_corners) of detected corners from the sample data for a try of calibration.

Optional

VTK =7.1.1: 3D Visualization
- for macOS:<br>
```
brew install vtk
```

Usage

Installation

git clone --recurse-submodules https://github.com/mfxox/ILCC
cd ILCC
python setup.py install

Explanation of files

config.py: parameter settings <br> img_corners_est.py: estimate corners of chessboard from images with OpenCV or MATLAB<br> pcd_corners_est.py: estimate corners of chessboard from the point cloud<br> LM_opt.py: load corresponding 2D-3D corners, calculate initial values with the PnP method, refine the result with LM method<br> utility.py: utility functions for various of visualization

Process data

Make a folder for example named as DATA and make the image and point cloud folders DATA/img and DATA/pcd respectively.
Put panoramic images into DATA/img and point cloud files into DATA/pcd. The files should be named like 00XX.png or 00XX.csv.
cd DATA and copy config.yaml to DATA and modify config.yaml according to your situation.
Corner detection from images.<br>
```
from ILCC import img_corners_est
img_corners_est.detect_img_corners()
```
Coordinates of corners from images are saved to DATA/output/img_corners with the filename 00XX_img_corners.txt and images with marked corners are saved in the same folder with the file name 00XX_detected_corners.jpg if 'output_img_with_dectected_corners' in config.yaml is set to True, as shown below.
<div style="text-align: center"> <img src="readme_files/0001_detected_corners.jpg" width = "50%" /> <img src="readme_files/0001_detected_corners_zoom.jpg" width = "24.35%" /> </div>
Corner detection from point clouds.<br>
```
from ILCC import pcd_corners_est
pcd_corners_est.detect_pcd_corners()
```
Coordinates of corners from point clouds are save to output/pcd_seg with the filename 00XX_pcd_result.pkl. Segments of each point cloud are output to /DATA/output/pcd_seg/00XX.
Non-linear optimization for final extrinsic parameters.<br>
```
from ILCC import LM_opt
LM_opt.cal_ext_paras()
```
The extrinsic calibration results are output in the end of the process and saved with the filename YYYYMMDD_HHMMSS_calir_result.txt. Images of back-projected 3D corners using the calculated parameters are saved to DATA/output if 'back_proj_corners' is set to True, as shown below.
<div style="text-align: center"> <img src="readme_files/0001_cal_backproj.jpg" width = "50%" /> <img src="readme_files/0001_cal_backproj_zoom.jpg" width = "24.35%" /><br>  </div>
After the aforementioned process, utility module can be imported for visualizing various of results. <br>
```
from ILCC import utility
utility.vis_back_proj(ind=1, img_style="orig", pcd_style="dis", hide_occlussion_by_marker=False)
utility.vis_back_proj(ind=1, img_style="orig", pcd_style="dis", hide_occlussion_by_marker=True)
utility.vis_back_proj(ind=1, img_style="edge", pcd_style="intens", hide_occlussion_by_marker=True)
```
The image (see below) with back-projected point cloud with the calculated extrinsic parameters will be showed and press "s" for saving. img_style can be "edge" (edge extracted) or "orig" (original image) and pcd_style can be "dis" (color by distance) or "intens" (color by intensity).
<div style="text-align: center"> <p align="center"> <img src="readme_files/0001_orig_dis.jpg" width = "80%" /><br> <em>Project points to the original image with coloring by distance. The occluded part by the chessboard is not hided.</em> <br> <img src="readme_files/0001_orig_dis_hide_occlusion.jpg" width = "80%" /><br> <em>The occluded part by the chessboard is hided. The occluded part by the chessboard is hided by setting the parameter hide_occlussion_by_marker True.</em> <br> Check the upper part of the chessboard in the two images above. </em> <br> <img src="readme_files/0001_edge_intens_hide_occlusion.jpg" width = "80%" /><br> <em>Project points to the edge image with coloring by intensity. Occluded points by the chessboard are hided.</em> </p> </div> <div style="text-align: center"> <p align="center"> <img src="readme_files/0001_orig_dis.png" width = "20%" /> <img src="readme_files/0001_orig_dis_hide_occlusion.png" width = "20%" /> <img src="readme_files/0001_edge_intens.png" width = "20%" /> <img src="readme_files/0001_edge_intens_hide_occlusion.png" width = "20%" /> <br> <em>Results of perspective images. From left to right: [color: distance, original image], [color: distance, original image, hide occlusion], [color: intensity, edge image], [color: intensity, edge image, hide occlusion]. </em> </p> </div>

For 3D visualization, [VTK](https:// clhub.com/Kitware/VTK) >=7.0 is necessary. See the example below for how to use.

Example

Sample Data

The sample data and processing results of detected corners can be downloaded from

ILCC

Install / Use

README