RING#: PR-by-PE Global Localization with Roto-translation Equivariant Gram Learning

Paper: Sha Lu, Xuecheng Xu, Dongkun Zhang, Yuxuan Wu, Haojian Lu, Xieyuanli Chen, Rong Xiong, and Yue Wang, "RING#: PR-by-PE Global Localization with Roto-translation Equivariant Gram Learning", IEEE Transactions on Robotics (TRO), 2025.

Maintainer: Sha Lu, lusha@zju.edu.cn

📌 Table of Contents

• Introduction
• Key Features
• Prerequisites
• Installation
• Data Preparation
• Training
• Evaluation
• Citation
• Acknowledgments
• License

🌍 Introduction <a name="introduction"></a>

RING# is an end-to-end, PR-by-PE (Place Recognition by Pose Estimation) global localization framework that operates on BEV (Bird's Eye View) features compatible with both vision and LiDAR sensors. RING# incorporates a novel design that learns two equivariant representations from BEV features, enabling globally convergent and computationally efficient pose estimation. Comprehensive experiments on the NCLT and Oxford datasets show that RING# outperforms state-of-the-art methods in both vision and LiDAR modalities, validating the effectiveness of the proposed approach.

<img src="imgs/overview.png" width="100%"/>

⚡ Key Features <a name="key-features"></a>

• End-to-End Framework: RING# is designed to be an end-to-end framework, allowing for seamless integration of all components.
• PR-by-PE Localization: Derives place recognition similarity directly from correlation-based pose estimation, reducing error accumulation compared to traditional PR-then-PE localization methods.
• Vision and LiDAR Compatibility: Supports both vision and LiDAR inputs by operating on BEV features, making it versatile for various robotic applications.
• Global Convergence: Learns equivariant representations that handle pose transformations (x, y, θ), ensuring convergence to the optimal solution without initialization.
• Computational Efficiency: Decouples pose estimation into rotation and translation, accelerated by Fast Fourier Transform (FFT) and GPU batch processing, enabling fast matching.

🛠️ Prerequisites <a name="prerequisites"></a>

• miniconda or anaconda
• CUDA (We have tested with CUDA 11.3 and 11.6)

🚀 Installation <a name="installation"></a>

1. Create a virtual environment:

   conda create -n ringsharp python=3.8
   conda activate ringsharp
   

2. Clone the repository and install core dependencies:

   It's highly recommended to use the exact versions specified in requirements.txt to ensure reproducibility.

   git clone https://github.com/lus6-Jenny/RINGSharp.git
   cd RINGSharp
   pip install -r requirements.txt
   

   (Note: If you encounter issues, check compatibility with your CUDA and PyTorch versions.)

3. Install external dependencies from source:

   Some components (fast_gicp, torch-radon) require manual compilation:

   # Install fast_gicp (if needed for point cloud registration)
   git clone https://github.com/SMRT-AIST/fast_gicp.git --recursive
   cd fast_gicp
   python setup.py install --user
   
   # Install torch-radon (required for RING#)
   git clone https://github.com/matteo-ronchetti/torch-radon.git -b v2
   cd torch-radon
   python setup.py install
   

4. Install the package:

   # Install custom CUDA ops for glnet
   cd glnet/ops
   python setup.py develop
   
   # Install the main glnet package
   cd ../..
   python setup.py develop
   

📂 Data Preparation <a name="data-preparation"></a>

1. Download datasets:

   • NCLT Dataset
   • Oxford Radar Robotcar Dataset

   Organize the downloaded data into directories. Example structure:

   ~/Data/
   ├── NCLT/
   │   ├── yyyy-mm-dd/ (e.g., 2012-01-08)
   │   │   ├── velodyne_sync/ (Velodyne data)
   │   │   │   ├── xxxxxx.bin
   │   │   │   ├── ...
   │   │   ├── ground_truth/ (Ground truth data) 
   │   │   │   ├── groundtruth_yyyy-mm-dd.csv (e.g., groundtruth_2012-01-08.csv)
   │   │   ├── lb3/ (Ladybug camera data)
   │   │   │   ├── Cam0/
   │   │   │   ├── Cam1/
   │   │   │   ├── Cam2/
   │   │   │   ├── Cam3/
   │   │   │   ├── Cam4/
   │   │   │   ├── Cam5/
   │   │   ├── cam_param/ (Camera parameters)
   │   │   ├── U2D/ (Distortion parameters)
   │   │   └── ...
   │   └── ...
   └── Oxford_radar/
       ├── yyyy-mm-dd-hh-mm-ss/ (e.g., 2019-01-11-13-24-51)
       │   ├── velodyne_left/ (Left Velodyne data)
       │   │   ├── xxxxxx.bin
       │   │   ├── ...
       │   ├── velodyne_right/ (Right Velodyne data)
       │   │   ├── xxxxxx.bin
       │   │   ├── ...
       │   ├── gps.csv
       │   ├── ins.csv
       │   ├── mono_left/
       │   ├── mono_rear/
       │   ├── mono_right/
       │   ├── stereo/
       │   │   ├── centre/
       ├── models/ (Camera parameters)
       │   └── ...
       ├── extrinsics/ (Extrinsic parameters)
       │   └── ...
       └── ...
   

   (Note: Adjust paths in config files according to your storage location.)

2. Preprocess images:

   For image data, you need to first preprocess the images (undistort, rectify, crop, resize, etc.) and save them in the appropriate format:

   # ------ NCLT Dataset ------
   cd glnet/datasets/nclt
   python image_preprocess.py # The generated images will be saved in the 'lb3_u_s_384/Cam{0,1,2,3,4,5}' directories in jpg format
   
   # ------ Oxford Radar Dataset ------
   cd glnet/datasets/oxford
   python image_preprocess.py # The generated images will be saved in the 'mono_left_rect', 'mono_rear_rect', 'mono_right_rect', 'stereo/centre_rect' directories in png format
   

3. Generate training and evaluation data:

   These scripts create .pickle files containing metadata, poses, and other information for training and evaluation:

   # ------ NCLT Dataset ------
   cd glnet/datasets/nclt
   # Generate training and validation pickles
   python generate_training_tuples.py --dataset_root ~/Data/NCLT
   # Generate evaluation pickle
   python generate_evaluation_sets.py --dataset_root ~/Data/NCLT
   
   # ------ Oxford Radar Dataset ------
   cd glnet/datasets/oxford
   # Generate training and validation pickles
   python generate_training_tuples.py --dataset_root ~/Data/Oxford_radar
   # Generate evaluation pickle
   python generate_evaluation_sets.py --dataset_root ~/Data/Oxford_radar
   
   # ------ Optional Data Generation ------
   # Generate bev images for LiDAR data (use --bev flag)
   python generate_training_tuples.py --dataset_root ~/Data/NCLT --bev
   # Generate panorama images for vision data (use --sph flag)
   python generate_training_tuples.py --dataset_root ~/Data/NCLT --sph # defaule scale of panorama size is 2
   

   (Note: The generated .pickle files will be saved in the dataset_root directory.)

4. Configure the dataset and training parameters:

   The configuration files are located in the glnet/config directory. You can use the provided example files as templates:

   • config_nclt.txt: Configuration for the NCLT dataset.
   • config_oxford.txt: Configuration for the Oxford Radar dataset.
   • ring_sharp_v_nclt.txt: Configuration for the RING#-V model for the NCLT dataset.
   • netvlad_pretrain_oxford.txt: Configuration for the NetVLAD model for the Oxford Radar dataset.
   • ...

   You can modify these files to suit your specific dataset and training parameters. For example, you can set the dataset type, root directory, evaluation set, loss function, batch size, learning rate, and other parameters.

🏗️ Training <a name="training"></a>

The training script is located in the tools directory. You can run the training script with the following command:

cd tools
python train.py --dataset_type nclt \
                --dataset_root ~/Data/NCLT \
                --exp_name ring_sharp_v_nclt_run1 \
                --config ../glnet/config/config_nclt.txt \
                --model_config ../glnet/config/ring_sharp_v_nclt.txt \
                # --resume \
                # --weight xxx.pth

Key flags:

• --dataset_type: Specify the dataset type (e.g., nclt, oxford).
• --dataset_root: Path to the dataset root directory.
• --exp_name: Name of the experiment (used for saving checkpoints).
• --config: Path to the configuration file.
• --model_config: Path to the model configuration file.
• --resume: Flag to resume training from a checkpoint.
• --weight: Path to the pre-trained model weights.

📊 Evaluation <a name="evaluation"></a>

1. Evaluate the model of RING# (RING#-V, RING#-L):
   • Pose estimation evaluation:

     cd tools
     python evaluate_ours_pe.py --dataset_type nclt \
                                --dataset_root ~/Data/NCLT \
                                --exp_name ring_sharp_v_nclt_run1 \