[CVPR 2026]From Detection to Association: Learning Discriminative Object Embeddings for Multi-Object Tracking

TL;DR. We reveal that DETR-based end-to-end MOT suffers from overly similar object embeddings. FDTA explicitly enhances discriminativeness in this paradigm.

📢 News

• [2026] Our paper has been accepted by CVPR 2026! The code is now released. Please star this repo for updates!

🚀 Getting Started

1. Environment Setup

conda create -n FDTA python=3.12
conda activate FDTA
pip install -r requirements.txt
# Compile the Deformable Attention operator:
cd models/ops/
sh make.sh

2. Data Preparation

Please organize your datasets under ./datasets/ in the following structure. Note that depth maps are required alongside the RGB images:

datasets/
├── DanceTrack/
│   ├── train/
│   │   └── <sequence>/
│   │       ├── img1/
│   │       ├── gt/
│   │       └── depth/
│   ├── val/
│   └── test/
├── SportsMOT/
│   ├── train/
│   │   └── <sequence>/
│   │       ├── img1/
│   │       ├── gt/
│   │       └── depth/
│   ├── val/
│   └── test/
└── BFT/
    ├── train/
    │   └── <sequence>/
    │       ├── img1/
    │       ├── gt/
    │       └── depth/
    └── test/

Dataset sources:

• DanceTrack: Download from the official repository.
• SportsMOT: Download from the official repository or HuggingFace.
• BFT: Download from Google Drive or Baidu Pan (NetTrack).
• Depth maps: Generated using Video-Depth-Anything (see tools/gen_depthmaps.py). Other depth estimators with the same directory structure are also supported.

3. Pre-trained Weights

We use COCO pre-trained Deformable DETR weights for initialization, sourced from MOTIP. Download the weights and place them under ./pretrains/:

| File | Description | Download | |------|-------------|----------| | r50_deformable_detr_coco.pth | COCO pre-trained (base) | link | | r50_deformable_detr_coco_dancetrack.pth | Fine-tuned on DanceTrack | link | | r50_deformable_detr_coco_sportsmot.pth | Fine-tuned on SportsMOT | link | | r50_deformable_detr_coco_bft.pth | Fine-tuned on BFT | link |

Our trained FDTA model weights are available on HuggingFace. Download and place them under ./checkpoints/ for inference.

4. Training

All training configurations are stored in the ./configs/ folder. For example, to train on DanceTrack:

accelerate launch --num_processes=4 train.py \
  --data-root /path/to/your/datasets/ \
  --exp-name fdta_dancetrack \
  --config-path ./configs/dancetrack.yaml \
  --detr-pretrain ./pretrains/r50_deformable_detr_coco_dancetrack.pth

Replace dancetrack with sportsmot or bft to train on other datasets.

Note: If your GPU memory is less than 24GB, you can set --detr-num-checkpoint-frames 2 (< 16GB) or --detr-num-checkpoint-frames 1 (< 12GB) to reduce memory usage.

5. Inference

We support two inference modes:

• submit: Generate tracker files for submission .
• evaluate: Generate tracker files and compute evaluation metrics.

accelerate launch --num_processes=4 submit_and_evaluate.py \
  --data-root /path/to/your/datasets/ \
  --inference-mode evaluate \
  --config-path ./configs/dancetrack.yaml \
  --inference-model ./checkpoints/dancetrack.pth \
  --outputs-dir ./outputs/ \
  --inference-dataset DanceTrack \
  --inference-split val

accelerate launch --num_processes=4 submit_and_evaluate.py \
  --data-root /path/to/your/datasets/ \
  --inference-mode submit \
  --config-path ./configs/dancetrack.yaml \
  --inference-model ./checkpoints/dancetrack.pth \
  --outputs-dir ./outputs/ \
  --inference-dataset DanceTrack \
  --inference-split test

You can add --inference-dtype FP16 for faster inference with minimal performance loss.

---

Main Results

DanceTrack

| Training Data | HOTA | IDF1 | AssA | MOTA | DetA | |---------------|------|------|------|------|------| | train | 71.7 | 77.2 | 63.5 | 91.3 | 81.0 | | train+val | 74.4 | 80.0 | 67.0 | 92.2 | 82.7 |

SportsMOT

| Training Data | HOTA | IDF1 | AssA | MOTA | DetA | |---------------|------|------|------|------|------| | train | 74.2 | 78.5 | 65.5 | 93.0 | 84.1 |

BFT

| Training Data | HOTA | IDF1 | AssA | MOTA | DetA | |---------------|------|------|------|------|------| | train | 72.2 | 84.2 | 74.5 | 78.2 | 70.1 |

Acknowledgements

The code is built on top of these awesome repositories. We thank the authors for opensourcing their code.

• Deformable-DETR
• MOTR
• MOTIP
• MonoDETR
• TrackEval

Citation

If you find our work useful for your research, please consider citing:

@article{shao2025fdta,
  title={From Detection to Association: Learning Discriminative Object Embeddings for Multi-Object Tracking},
  author={Shao, Yuqing and Yang, Yuchen and Yu, Rui and Li, Weilong and Guo, Xu and Yan, Huaicheng and Wang, Wei and Sun, Xiao},
  journal={arXiv preprint arXiv:2512.02392},
  year={2025}
}