LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry

[CVPR 2024] The repository contains the official implementation of LEAP-VO. We aim to leverage temporal context with long-term point tracking to achieve motion estimation, occlusion handling, and track probability modeling.

LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry<br> Weirong Chen, Le Chen, Rui Wang, Marc Pollefeys<br> CVPR 2024

[Paper] [Project Page]

<div align="center"> <p align="center"> <a href=""> <img src="./assets/demo.gif" alt="Logo" width="70%"> </a> </p> </div>

Installation

Requirements

The code was tested on Ubuntu 20.04, PyTorch 1.12.0, CUDA 11.3 with 1 NVIDIA GPU (RTX A4000).

Clone the repo

git clone https://github.com/wrchen530/leapvo.git
cd leapvo 

Create a conda environment

conda env create -f environment.yml
conda activate leapvo

Install LEAP-VO package

wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip
unzip eigen-3.4.0.zip -d thirdparty

pip install .

Demos

Our method requires an RGB video and camera intrinsics as input. We provide the model checkpoint and example data on Google Drive. Please download leap_kernel.pth and place it in the weights folder, and download samples and place them in the data folder.

The demo can be run using:

python main/eval.py \
  --config-path=../configs \
  --config-name=demo \                                  # config file
  data.imagedir=data/samples/sintel_market_5/frames \   # path to image directory or video
  data.calib=data/samples/sintel_market_5/calib.txt \   # calibration file
  data.savedir=logs/sintel_market_5 \                   # save directory
  data.name=sintel_market_5 \                           # scene name
  save_trajectory=true \                                # save trajectory in TUM format
  save_video=true \                                     # save video visualization
  save_plot=true                                        # save trajectory plot

Evaluations

We provide evaluation scripts for MPI-Sinel, TartanAir-Shibuya, and Replica.

MPI-Sintel

Follow MPI-Sintel and download it to the data folder. For evaluation, we also need to download the groundtruth camera pose data. The folder structure should look like

MPI-Sintel-complete
└── training
    ├── final
    └── camdata_left

Then run the evaluation script after setting the DATASET variable to custom location.

bash scripts/eval_sintel.sh

TartanAir-Shibuya

Follow TartanAir-Shibuya and download it to the data folder. Then run the evaluation script after setting the DATASET variable to custom location.

bash scripts/eval_shibuya.sh

Replica

Follow Semantic-NeRF and download the Replica dataset into data folder. Then run the evaluation script after setting the DATASET variable to custom location.

bash scripts/eval_replica.sh

Citations

If you find our repository useful, please consider citing our paper in your work:

@InProceedings{chen2024leap,
  title={LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry},
  author={Chen, Weirong and Chen, Le and Wang, Rui and Pollefeys, Marc},
  journal={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2024}
}

Acknowledgement

We adapted some codes from some awesome repositories including CoTracker, DPVO, and ParticleSfM. We sincerely thank the authors for open-sourcing their work and follow the License of CoTracker, DPVO and ParticleSfM.