LoftUp: A Coordinate-Based Feature Upsampler for Vision Foundation Models

ICCV2025 (oral)

[Arxiv] [Project Page]

Haiwen Huang, Anpei Chen, Volodymyr Havrylov, Andreas Geiger, Dan Zhang

TL;DR: LoftUp achieves the strongest feature upsampling performance at a comparable speed to bilinear upsampling.

Contents

• Install
• Inference with pretrained upsamplers
• Evaluation on downstream tasks
• Training LoftUp upsamplers
• Citation

Install

In general, LoftUp can run with most recent pytorch environments. We encourage the users to try out LoftUp in their exisitng environment first.

We also provide two yaml file for installation. To use them, simply run:

conda env create -f environment_cuda11.yaml

or

conda env create -f environment.yaml

Inference with pretrained upsamplers

All pre-trained upsamplers are available on 🤗 here: https://huggingface.co/models?search=loftup.

We provide example code for using LoftUp in example_usage.py. Currently we provide:

|Backbone Name | Featurizer Class | HF hub | Torch Hub Repo | Torch Hub Name | |-------------------| ---|------------------------------------------------|------|-----| | DINOv2 S/14 | dinov2 | haiwen/loftup-dinov2s | andrehuang/loftup | loftup_dinov2s| | DINOv2 S/14 + Reg | dinov2s_reg | haiwen/loftup-dinov2s_reg| andrehuang/loftup | loftup_dinov2s_reg| | DINOv2 B/14 | dinov2b | haiwen/loftup-dinov2b | andrehuang/loftup | loftup_dinov2b| | DINOv2 B/14 + Reg | dinov2b_reg | haiwen/loftup-dinov2b_reg|andrehuang/loftup | loftup_dinov2b_reg| | CLIP ViT B/16 | clip |haiwen/loftup-clip | andrehuang/loftup | loftup_clip| |SigLIP ViT B/16 | siglip | haiwen/loftup-siglip| andrehuang/loftup | loftup_siglip| |SigLIP2 ViT B/16 | siglip2 | haiwen/loftup-siglip2| andrehuang/loftup | loftup_siglip2|

To use torch hub checkpoints, simply run

upsampler = torch.hub.load('andrehuang/loftup', model_torch_hub_name, pretrained=True)

For example, upsampler = torch.hub.load('andrehuang/loftup', loftup_dinov2s, pretrained=True).

The upsampler class is defined at UpsamplerwithChannelNorm.

Evaluation on Downstream Tasks

Dataset Preparation

See Preparing Datasets for Evaluation.

Semantic Segmentation

For semantic segmentation, our implementation is adapted from FeatUp. You can use eval_seg.py by running:

python eval_seg.py  ++upsampler_path=/path/to/your/upsampler

You can also configure other hyper-parameters such as output_dir and dataset directory. The config file is configs/eval_seg.yaml.

Video Object Segmentation

For video object segmentation on DAVIS, our code is modified from the implementation in LiFT. Specifically, we first extract segmentaiton results by running:

    python eval_davis.py --dataroot your_davis_data_dir --model_type "dinov2" --output_dir your_output_dir --imsize 224 --upsampler_path=your_upsampler_path

Then run the following to get evaluation results:

python davis2017-evaluation/evaluation_method.py --davis_path /your_davis_data_dir --task semi-supervised --results_path your_output_dir/davis_vidseg_224 --imsize 224

Others

For interactive segmentation, please check out iSegProbe.

For open-vocabulary segmentation, please check out ProxyCLIP.

For depth and normal estimation, please check out Probe3D.

Training LoftUp Upsamplers

This repository contains training scripts for training LoftUp upsamplers. The training is done in two stages:

Stage 1: Basic Feature Upsampling

Stage 1 training (train_loftup_stage1.py) trains upsamplers to convert low-resolution features to high-resolution features using reconstruction loss.

Example training command:

python train_loftup_stage1.py ++dataset="sa1b" ++epochs=1 ++batch_size=2 ++num_gpus=4 ++model_type="dinov2" ++pytorch_data_dir='datasets' ++upsampler_type="loftup" ++sam_mask_alpha=0.8 ++load_size=224 ++upsample_size=224 ++tv_weight=0.001 ++clamp_featup=True

Stage 2: High-Resolution Supervision

Stage 2 training (train_loftup_stage2.py) fine-tunes the Stage 1 upsampler with high-resolution supervision for improved quality.

Example training command:

python train_loftup_stage2.py ++dataset="sa1b" ++epochs=1 ++hr_res=896 ++batch_size=2 ++consistency_method="bilinear" ++model_type="dinov2" ++num_gpus=4 ++affinity_loss=True ++pytorch_data_dir='datasets' ++pretrained_upsampler="path/to/stage1_checkpoint.ckpt" ++upsampler_type="loftup" ++sam_mask_hr_alpha=0.5 ++sam_mask_reg=0.0 ++lr=1e-3 ++use_featup=False ++aug_size ++n_jitters=2

Configuration

Both training scripts use Hydra for configuration management. Configuration files are located in configs/:

• configs/train_loftup_stage1.yaml - Stage 1 configuration
• configs/train_loftup_stage2.yaml - Stage 2 configuration

Key configuration parameters:

• model_type: Feature extractor type (e.g., "dinov2", "clip")
• upsampler_type: Type of upsampler to train (e.g., "loftup")
• batch_size: Training batch size
• epochs: Number of training epochs
• lr: Learning rate
• load_size: Input image size for feature extraction
• upsample_size: Target size for upsampled features
• n_jitters: Number of jittering augmentations per training step
• tv_weight: Weight for total variation loss
• sam_mask_alpha: Weight for SAM mask adjustment (Stage 1)
• sam_mask_hr_alpha: Weight for SAM mask adjustment (Stage 2)

For more details, see the configuration files in configs/ and the training scripts themselves.

Citation

If you find our work helpful, please cite:

@misc{huang2025loftuplearningcoordinatebasedfeature,
      title={LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models}, 
      author={Haiwen Huang and Anpei Chen and Volodymyr Havrylov and Andreas Geiger and Dan Zhang},
      year={2025},
      eprint={2504.14032},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2504.14032}, 
}