Mast3r

Official implementation of Grounding Image Matching in 3D with MASt3R
[Project page], [MASt3R arxiv], [DUSt3R arxiv]

@misc{mast3r_eccv24,
      title={Grounding Image Matching in 3D with MASt3R}, 
      author={Vincent Leroy and Yohann Cabon and Jerome Revaud},
      booktitle = {ECCV},
      year = {2024}
}

@misc{mast3r_arxiv24,
      title={Grounding Image Matching in 3D with MASt3R}, 
      author={Vincent Leroy and Yohann Cabon and Jerome Revaud},
      year={2024},
      eprint={2406.09756},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

@inproceedings{dust3r_cvpr24,
      title={DUSt3R: Geometric 3D Vision Made Easy}, 
      author={Shuzhe Wang and Vincent Leroy and Yohann Cabon and Boris Chidlovskii and Jerome Revaud},
      booktitle = {CVPR},
      year = {2024}
}

@inproceedings{
    duisterhof2025mastrsfm,
    title={{MAS}t3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motion},
    author={Bardienus Pieter Duisterhof and Lojze Zust and Philippe Weinzaepfel and Vincent Leroy and Yohann Cabon and Jerome Revaud},
    booktitle={International Conference on 3D Vision 2025},
    year={2025},
    url={https://openreview.net/forum?id=5uw1GRBFoT}
} 

Table of Contents

• Table of Contents
• License
• Get Started
  • Installation
  • Checkpoints
    • MASt3R Model
    • Retrieval Model
    • Dune Model
  • MASt3R-SfM
  • Interactive demo
  • Interactive demo with docker
• Usage
  • Usage MASt3R
  • Usage DUNE+MASt3R
• Training
  • Datasets
  • Demo
  • Our Hyperparameters
• Visual Localization
  • Dataset preparation
  • Example Commands

License

The code is distributed under the CC BY-NC-SA 4.0 License. See LICENSE for more information.

# Copyright (C) 2024-present Naver Corporation. All rights reserved.
# Licensed under CC BY-NC-SA 4.0 (non-commercial use only).

Get Started

Installation

1. Clone MASt3R.

git clone --recursive https://github.com/naver/mast3r
cd mast3r
# if you have already cloned mast3r:
# git submodule update --init --recursive

2. Create the environment, here we show an example using conda.

conda create -n mast3r python=3.11 cmake=3.14.0
conda activate mast3r 
conda install pytorch torchvision pytorch-cuda=12.1 -c pytorch -c nvidia  # use the correct version of cuda for your system
pip install -r requirements.txt
pip install -r dust3r/requirements.txt
# Optional: you can also install additional packages to:
# - add support for HEIC images
# - add required packages for visloc.py
pip install -r dust3r/requirements_optional.txt

3. compile and install ASMK

pip install cython

git clone https://github.com/jenicek/asmk
cd asmk/cython/
cythonize *.pyx
cd ..
pip install .  # or python3 setup.py build_ext --inplace
cd ..

4. Optional, compile the cuda kernels for RoPE (as in CroCo v2).

# DUST3R relies on RoPE positional embeddings for which you can compile some cuda kernels for faster runtime.
cd dust3r/croco/models/curope/
python setup.py build_ext --inplace
cd ../../../../

Checkpoints

MASt3R Model

You can obtain the model checkpoints by two ways:

1. You can use our huggingface_hub integration: the models will be downloaded automatically.

2. Otherwise, download it from our server:

| Modelname | Training resolutions | Head | Encoder | Decoder | |-------------|----------------------|------|---------|---------| | MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric | 512x384, 512x336, 512x288, 512x256, 512x160 | CatMLP+DPT | ViT-L | ViT-B |

You can check the hyperparameters we used to train these models in the section: Our Hyperparameters Make sure to check license of the datasets we used.

To download MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth:

mkdir -p checkpoints/
wget https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth -P checkpoints/

Make sure to agree to the license of all the training datasets we used, in addition to CC-BY-NC-SA 4.0. The mapfree dataset license in particular is very restrictive. For more information, check CHECKPOINTS_NOTICE.

Retrieval Model

This retrieval model is for MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric only. You need to download both the trainingfree.pth and codebook.pkl files, and put them in the same directory.

MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric_retrieval_trainingfree
MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric_retrieval_codebook

mkdir -p checkpoints/
wget https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric_retrieval_trainingfree.pth -P checkpoints/
wget https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric_retrieval_codebook.pkl -P checkpoints/

Dune Model

We added partial support of the Dune encoder. Check the associated Dune License.
You can find the MASt3R decoder that goes with it here:

dunemast3r_cvpr25_vitbase
dunemast3r_cvpr25_vitsmall

mkdir -p checkpoints/
wget https://download.europe.naverlabs.com/dune/dunemast3r_cvpr25_vitbase.pth -P checkpoints/

This model have limited compatility with the rest of the codebase, but we wanted to include it as it achieves impressive results on the Map-free Visual Relocalization benchmark. Make sure to check the Usage DUNE+MASt3R section if you are interested.

MASt3R-SfM

A few words about the addition of MASt3R-SfM to this repository.

MASt3R-SfM refers to the make_pairs (retrieval) + sparse_global_alignment that you can find here: demo.py#L142.

In this repository, you will also find kapture_mast3r_mapping.py and demo_glomap.py. These two scripts are unrelated to MASt3R-SfM. They are "toys" that attempt to use mast3r matches to do standard Sfm reconstructions with colmap/glomap. As such, they were not extensively tested, and may fail on edge cases.

Interactive demo

We made one huggingface space running the new sparse global alignment in a simplified demo for small scenes: naver/MASt3R There are two demos available to run locally:

demo.py is the updated demo for MASt3R. It uses our new sparse global alignment method that allows you to reconstruct larger scenes

python3 demo.py --model_name MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric

# Use --weights to load a checkpoint from a local file, eg --weights checkpoints/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth
# Use --retrieval_model and point to the retrieval checkpoint (*trainingfree.pth) to enable retrieval as a pairing strategy, asmk must be installed
# Use --local_network to make it accessible on the local network, or --server_name to specify the url manually
# Use --server_port to change the port, by default it will search for an available port starting at 7860
# Use --device to use a different device, by default it's "cuda"

demo_dust3r_ga.py is the same demo as in dust3r (+ compatibility for MASt3R models)
see https://github.com/naver/dust3r?tab=readme-ov-file#interactive-demo for details

Interactive demo with docker

TODO update with asmk/retrieval model

To run MASt3R using Docker, including with NVIDIA CUDA support, follow these instructions:

1. Install Docker: If not already installed, download and install docker and docker compose from the Docker website.

2. Install NVIDIA Docker Toolkit: For GPU support, install the NVIDIA Docker toolkit from the Nvidia website.

3. Build the Docker image and run it: cd into the ./docker directory and run the following commands:

cd docker
bash run.sh --with-cuda --model_name="MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric"

Or if you want to run the demo without CUDA support, run the following command:

cd docker
bash run.sh --model_name="MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric"

By default, demo.py is launched with the option --local_network.
Visit http://localhost:7860/ to access the web UI (or replace localhost with the machine's name to access it from the network).

run.sh will launch docker-compose using either the docker-compose-cuda.yml or docker-compose-cpu.ym config file, then it starts th