🚀 MonSter (CVPR 2025 Highlight) 🚀

<p align="center"> </p> <h2>🤗 Demo Video：</h2> <div align="center"> <a href="https://www.youtube.com/watch?v=u-UhFMGmXro"> <img src="media/video.png" width="50%" alt="Overview Video"> </a> </div> <p></p> <div align="center"> </div> MonSter: Marry Monodepth to Stereo Unleashes Power. <a href="https://arxiv.org/abs/2501.08643"><img src="https://img.shields.io/badge/arXiv-2501.08643-b31b1b?logo=arxiv" alt='arxiv'></a> <a href='https://openbayes.com/console/public/tutorials/IHQPgJwkqUh'><img src='https://img.shields.io/badge/Demo-OpenBayes贝式计算-blue'></a>

Junda Cheng, Longliang Liu, Gangwei Xu, Xianqi Wang, Zhaoxing Zhang, Yong Deng, Jinliang Zang, Yurui Chen, Zhipeng Cai, Xin Yang<br/>

News

[2025/9] We have open-sourced our lightweight real-time model, RT-MonSter++, which is now available at https://github.com/Junda24/MonSter-plusplus
[2025/9] Weights for RT-MonSter++ model released!
[2025/10] Weights for MonSter++ model released!
[2025/10] We found a bug in the dataloader code of our sceneflow dataset. This should be a misoperation during code organization, but it has now been fixed.

🌼 Abstract

MonSter is an innovative approach that effectively harnesses the complementary strengths of monocular depth estimation and stereo matching, thereby fully unlocking the potential of stereo vision. It significantly enhances the depth perception performance of stereo matching in challenging regions such as ill-posed areas and fine structures. Notably, MonSter ranks first across five of the most widely used leaderboards, including SceneFlow, KITTI 2012, KITTI 2015, Middlebury, and ETH3D. Additionally, in terms of zero-shot generalization, MonSter also significantly and consistently outperforms state-of-the-art methods, making it the current model with the best accuracy and generalization capabilities.

✅ TODO List

[x] The real-time version of MonSter.
[x] Mobile-friendly versions of MonSter (e.g. drones).
[x] The multi-view version of MonSter (MVS).

🌈: Zero-shot performance on KITTI

visualization1 Zero-shot generalization performance on KITTI benchmark.

<img src="media/kitti.gif" width="1241" height="752"> video demo

:art: Zero-shot performance on the wild captured stereo images

visualization2 Zero-shot generalization performance on our captured stereo images.

📝 Benchmarks performance

teaser Comparisons with state-of-the-art stereo methods across five of the most widely used benchmarks.

⚙️ Installation

NVIDIA RTX 3090
python 3.8

⏳ Create a virtual environment and activate it.

conda create -n monster python=3.8
conda activate monster

🎬 Dependencies

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
pip install tqdm
pip install scipy
pip install opencv-python
pip install scikit-image
pip install tensorboard
pip install matplotlib 
pip install timm==0.6.13
pip install mmcv==2.1.0 -f https://download.openmmlab.com/mmcv/dist/cu118/torch2.1/index.html
pip install accelerate==1.0.1
pip install gradio_imageslider
pip install gradio==4.29.0

✏️ Required Data

✈️ Model weights

The mix_all model is trained on all the datasets mentioned above, which has the best performance on zero-shot generalization. There is a bug in the sceneflow model reproduction. We are working on it and will re-upload it later.

✈️ Evaluation

To evaluate the zero-shot performance of MonSter on Scene Flow, KITTI, ETH3D, vkitti, DrivingStereo, or Middlebury, run

python evaluate_stereo.py --restore_ckpt ./pretrained/sceneflow.pth --dataset *(select one of ["eth3d", "kitti", "sceneflow", "vkitti", "driving"])

or use the model trained on all datasets, which is better for zero-shot generalization.

python evaluate_stereo.py --restore_ckpt ./pretrained/mix_all.pth --dataset *(select one of ["eth3d", "kitti", "sceneflow", "vkitti", "driving"])

✈️ Submission

For MonSter submission to the KITTI benchmark, run

python save_disp.py

For MonSter submission to the Middlebury benchmark, run

python save_pfm.py

For MonSter submission to the ETH3D benchmark, run

python save_pfm_eth.py

✈️ Training

To train MonSter on Scene Flow or KITTI or ETH3D or Middlebury, run

CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch train_kitti.py   (for KITTI)
CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch train_eth3d.py   (for ETH3D)
CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch train_sceneflow.py   (for Scene Flow)
CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch train_middlebury.py   (for Middlebury)

✈️ FAQ

Q: Missing keys when using mix_all.pth

A: I updated the save_disp.py file and solved this problem.

✈️ Citation

If you find our works useful in your research, please consider citing our papers:


@InProceedings{Cheng_2025_CVPR,
    author    = {Cheng, Junda and Liu, Longliang and Xu, Gangwei and Wang, Xianqi and Zhang, Zhaoxing and Deng, Yong and Zang, Jinliang and Chen, Yurui and Cai, Zhipeng and Yang, Xin},
    title     = {MonSter: Marry Monodepth to Stereo Unleashes Power},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {6273-6282}
}

Acknowledgements

This project is based on RAFT-Stereo, GMStereo, and IGEV. We thank the original authors for their excellent works.

MonSter

Install / Use

README