LiCaS3

Here's the official implementation of the Paper: K. Yuan, L. Ding, M. Abdelfattah and Z. J. Wang, "LiCaS3: A Simple LiDAR–Camera Self-Supervised Synchronization Method," in IEEE Transactions on Robotics, doi: 10.1109/TRO.2022.3167455.

(speed x 8)

Project Walk-Thru

This project consists of four folders/parts:

• [X] training data generation / serialization
• [X] training
• [X] evaluation
  • evaluation over datasets
  • evaluation with limo
• [X] complementary newer college datasets

Each folder is independent so that you can run them separately. You can find the details of training data generation and training in this README while the evaluation has its own READMEs, since some evaluation requires quite a lot of setups. Demonstrative pre-trained weights are provided, and please refer to the README.md in evaluation folder. Raw results of evaluation with limo are also provided for whoever would like to compare my results and plot it on the same figure, referring to the evaluation with limo sub-folder.

Hardware Requirements

I conducted my experiments with the following hardware setups, which is highly recommended but not necessarily a hard requirement.

| Item | Version| |---|---| | System | 18.04 (x86), referring to MACHINE-SETUP.md to install the system| | SSD | 2T | | GPU Memory | >= 11 GB |

Main Software Requirements

| Item | Version | |---|----------------| | Python | 3.7.x | | NVIDIA-Driver | 440.36 | | CUDA | 10.2 | | cuDNN| v7.4.2 | | Tensorflow-GPU | >=1.14 & < 2 | | Opencv | 4.2 |

I use anaconda to manage virtual environment and except for NVIDIA-Driver and CUDA, all other dependencies are mostly used for training data generation. All training happens inside docker so that the environment is transferable.

Prepare Raw Datasets

KITTI

OK, first of all, you shall download the KITTI datasets:

• [X] Navigate to training_data_serialization
• [X] Copy bash script to where you wanna download the data (1-2 TB SSD recommended)
• [X] Run the bash bash download.sh to download the sequence 26_09 to your file dir (and this may take a while, like, a really long while depending on your network speeeeeed)
• [X] Manually split the training and testing datasets, and here's how my folder setup look like (tree $(YOUR_FOLDER}/kitti/raw/ -L 3):

The overall size of the raw data is fairly small, 62.3 GB.

├── test (8.3 GB)
│   └── 2011_09_26
│       ├── 2011_09_26_drive_0005_sync
│       ├── 2011_09_26_drive_0070_sync
│       ├── calib_cam_to_cam.txt
│       ├── calib_imu_to_velo.txt
│       └── calib_velo_to_cam.txt
└── train (53 GB)
    ├── 2011_09_26
    │   ├── 2011_09_26_drive_0001_sync
    │   ├── 2011_09_26_drive_0002_sync
    │   ├── .......
    │   ├── 2011_09_26_drive_0117_sync
    │   ├── calib_cam_to_cam.txt
    │   ├── calib_imu_to_velo.txt
    │   └── calib_velo_to_cam.txt
    └── download.sh

Newer College

You will need to request the download from this link:https://ori-drs.github.io/newer-college-dataset/

And I organize my folder like below, which is 38.3 GB in total.

├── raw_data
│   └── infra1 (45, 880 items, 9.2 GB)
    │   ├── infra1_1583836591_152386717.png
    │   ├── ......
│   └── ouster_scan (15, 302 items, 29.1 GB)
    │   ├── cloud_1583836591_182590976.pcd
    │   ├── ......
│   └── timeoffset
    │   ├── nc-short-time-offsets.csv

Generate/Serialize Training Data

Note: all below happens on your host machine instead of the docker container, mainly due to visualization requirements.

Depending on the stride, window, down-sample ratio, the volume of the datasets will really vary. To be safe, I just recommend you get a 1 TB SSD, which is what I used.

Code Walk-thru

Since we stick to tensorflow 1.x, all training data are created before-hand as a separate process in tfrecord format.

Below is the tree of the subfolder training_data_serialization, which is all you need to create the tfrecords:

training_data_serialization/
├── configs
│   ├── kitti.yaml
│   └── newer_college.yaml
├── data_generation_base.py
├── download.sh
├── __init__.py
├── kitti_data_generator.py
├── Makefile
├── newer_college_data_generator.py
├── newer_college_labelling_data_genearator.py
└── utils
    ├── __init__.py
    ├── kitti_loader.py
    ├── projection.py
    └── reduce_lidar_lines.py

It simply contains 3 main parts: generator scripts, config yamls and utils. utils is simply a folder containing helper functions of projections, data loader or reduce lidar lines. We only have two configs for kitti and newer college respectively. Beside training data configs, a redundant sensor information is also included for references. Finally, we have two data generator, which all are inherited from the Generator class in data_generator_base.py. As you can see from the class, regardless of the logics, the main flow is just three steps: load config, generate training data, serialize into tfrecords on disk. I also provided the script (newer_college_labelling_data_generator.py)) that I used to create the annotation datasets (which I used for benchmark only). You pbbly don't need this but I still put it here for references.

KITTI

First, take a look at the kitti.yaml:

Below are all fields that you will need to update before creating the tfrecords:

raw_data:
  ...
  root_dir: "Replace this with your training / validation / testing raw data sub-folder"

training_data:
  ...
  output_dir: "Replace this with where you would like to save the tfrecords"
  name:  "training" #  ["training", "testing", "validation"]
  sampling_window: 5 # by default, we use a single lidar frame
  sampling_stride: 1 # skip every every two frames
  features:
    X:
      ...
      C: 24 # (sampling_window + 1) * 4

It shall be noted that sampling_window here corresponds to l-1 in the paper. This is because computer starts from 0.

The comments shall explain itself very well. After this is done, just with one command as below, you shall be able to see the data generation running on multi-processors:

# navigate to the subfolder training_data_serialization
make create-kitti-data

Unfortunately, you will need to run training/validation/testing data generation separately. However, you can simply open three terminals and update the yaml on-the-fly to do the job.

You can tweak the following based on your compute power.

training_data:
  chunk_size: 5000
  downsample_ratio: 0.5 # TODO: This depends on how big the machine is

It will take a while to finish the training data generation. Taking sample window as 5, sampling stride as 1, downsample_ratio as 0.1 as an example, on my Intel Core i9-9900K CPU (16 cores, 32 GB Memory), it will take around 15 mins to serialize a 6.4 GB (2 files in total) training data.

Newer College

Like-wise, you will need to update similar fields in the newer_college.yaml:

raw_data:
  ...
  root_dir: "Replace this with your raw data folder"
  ...
  time_offsets_csv_fp: "Replace this with your timeoffset csv path" # timeoffset/nc-short-time-offsets.csv
  generated_fp:
    synced_raw_data_info: "Replace this with where you would like to save the synced raw data info" # xxxx/synced_raw_data_info.pkl
training_data:
  output_dir: "Replace this with where you would like to save the tfrecords"
  sampling_stride: 1
  sampling_window: 5
  features:
    X:
      ...
      C: 24  # (sampling_window + 1) * 4

The main difference is that you don't have to run the scripts multiple times manually for training/validation/testing since the split ratio does the trick. Note: the time_offsets_csv_fp is only used for metrics and the baseline (not really ground truth since it's quite poor) offset is not used as supervising signal.

# navigate to the subfolder training_data_serialization
make create-newer-college-data

Training

Note: all below happens inside the docker container, so that you can pbbly run it everywhere.

Build the Image and Enter a Shell

Build the docker image:

make build

Update the volume mount of your training data in the Makefile:

run:
	sudo docker run -it \
        --runtime=nvidia \
        --name="licas3-experiment" \
        --net=host \
        --privileged=true \
        --ipc=host \
        --memory="10g" \
        --memory-swap="10g" \
        -v ${PWD}:/root/licas3 \
        -v ${PWD}/newer_college:/root/newer_college \
        -v ${PWD}/kitti:/root/kitti \
      	licas3-docker bash

I by default believe that you put newer_college and kitti data under ${PWD}. However, if that's not true, update it.

At last, enter a shell with simply doing the following:

make run

And then you shall enter a shell with exact same environment of what I was using!

Experimental Data Management Suggestions

In you are working on a paper based on my works, which also needs to do ablation studies with various stride/window settings, I highly recommend you organize your training/validation/testing datasets as follows (taking kitti as an example):

kitti
├── raw
├── training_datasets
│   ├── sample_01_stride_05_beam_64_downsample_05/