Single/Multiple Object Tracking and Segmentation

Codes and comparison of recent single/multiple object tracking and segmentation.

News

:boom: VLT_SCAR/VLT_TT is accepted by NeurIPS2022.

:boom: CNNInMo/TransInMo is accepted by IJCAI2022.

:boom: CSTrack is accepted by IEEE TIP.

:boom: OMC is accepted by AAAI2022. The training and testing code has been released in this codebase.

:boom: AutoMatch is accepted by ICCV2021. The training and testing code has been released in this codebase.

:boom: CSTrack ranks 5/4000 at Tianchi Global AI Competition.

:boom: Ocean is accepted by ECCV2020. [OceanPlus] is accepted by IEEE TIP.

:boom: SiamDW is accepted by CVPR2019 and selected as oral presentation.

<!-- :boom: The improved version of [CSTrack_panda](https://github.com/JudasDie/SOTS/blob/master/lib/tutorial/CSTrack_panda/CSTrack_PANDA.md) has been released, containing the end-to-end tranining codes on PANDA. It is a strong baseline for [Gigavison](http://gigavision.cn/index.html) MOT tracking. Our tracker takes the **5th** place in **Tianchi Global AI Competition (天池—全球人工智能技术创新大赛[赛道二])**, with the score of **A-0.6712/B-0.6251 (AB榜)**, which surprisingly outperforms the baseline tracker JDE with score of A-0.32/B-0.34. More details about CSTrack_panda can be found [here](https://blog.csdn.net/qq_34919792/article/details/116792954?spm=1001.2014.3001.5501). --> <!-- [](https://www.youtube.com/watch?v=zRCRgsrW71s "") -->

Supported Trackers (SOT and MOT)

Single-Object Tracking (SOT)

• [x] [NeurIPS2022] VLT_SCAR/VLT_TT
• [x] [IJCAI2022] CNNInMo/TransInMo
• [x] [ICCV2021] AutoMatch
• [x] [ECCV2020] Ocean and Ocean+
• [x] [CVPR2019 Oral] SiamDW

Multi-Object Tracking (MOT)

• [x] [AAAI2022] OMC
• [x] [IEEE TIP] CSTrack

Results Comparison

• [x] Comparison

Branches

• SOT (or master): for our SOT trackers
• MOT: for our MOT trackers
• v0: old codebase supporting OceanPlus and TensorRT testing.

Please clone the branch to your needs.

Structure

• experiments: training and testing settings
• demo: figures for readme
• dataset: testing dataset
• data: training dataset
• lib: core scripts for all trackers
• snapshot: pre-trained models
• pretrain: models trained on ImageNet (for training)
• tracking: training and testing interface

$SOTS
|—— experimnets
|—— lib
|—— snapshot
  |—— xxx.model
|—— dataset
  |—— VOT2019.json 
  |—— VOT2019
     |—— ants1...
  |—— VOT2020
     |—— ants1...
|—— ...

Performance

| _Model</br> | _OTB2015</br> | _GOT10K</br> | _LaSOT</br> | _TNL2K</br> | _{TrackingNet</br>} | _NFS30</br> | _TOTB |_VOT2019 |_TC128 |_UAV123 |_{LaSOT_Ext} |_OTB-99-LANG | |:-----:|:-:|:----:|:------:|:--------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:| | <sub>SiamDW<sub> | _0.670 | _0.429 | _0.386|_0.348|_61.1| _0.521 |_0.500 |_0.241 |_0.583 |_0.536 |_- |_- | | _Ocean | _0.676 | _0.615 | _0.517|_0.421|_69.2| _0.553 |_0.638 |_0.323 |_0.585 |_0.621 |_- |_- | | _AutoMatch | _0.714 | _0.652 | _0.583|_0.472|_76.0| _0.606 |_0.668 |_0.322 |_0.634 |_0.644 |_- |_- | | _CNNInMo | _0.703 | _- | _0.539|_0.422|_72.1| _0.560 |_- |_- |_- |_0.629 |_- |_- | | _TransInMo | _0.711 | _- | _0.657|_0.520|_81.7| _0.668 |_- |_- |_- |_0.690 |_- |_- | | _{VLT_SCAR} | _- | _0.610 | _0.639|_0.498|_-| _- |_- |_- |_- |_- |_0.447 |_0.739 | | _{VLT_TT} | _- | _0.694 | _0.673|_0.531|_-| _- |_- |_- |_- |_- |_0.484 |_0.764 |

Tracker Details

VLT_SCAR/VLT_TT [NeurIPS2022]

[Paper] [Raw Results] [Training and Testing Tutorial] <br/> VLT explores a different path to achieve SOTA tracking without complex Transformer, i.e., multimodal Vision-Language tracking. The essence is a unified-adaptive Vision-Language representation, learned by the proposed ModaMixer and asymmetrical networks. The experiments show our approach surprisingly boosts a pure CNN-based Siamese tracker to achieve competitive or even better performances compared to recent SOTAs, which also benefits Transformer-based trackers. We hope that this work inspires more possibilities for future tracking beyond Transformer.

<img src="https://github.com/JudasDie/SOTS/blob/SOT/demo/VLT.jpg" width="700" alt="VLT"/><br/>

CNNInMo/TransInMo [IJCAI2022]

[Paper] [Raw Results] [Training and Testing Tutorial] <br/> CNNInMo/TransInMo introduces a novel mechanism that conducts branch-wise interactions inside the visual tracking backbone network (InBN) via the proposed general interaction modeler (GIM). We show that both CNN and Transformer backbones can benefit from InBN, with which more robust feature representation can be learned. Our method achieves compelling tracking performance by applying the backbones to Siamese tracking.

<img src="https://github.com/JudasDie/SOTS/blob/SOT/demo/TransInMo.jpg" width="700" alt="TransInMo"/><br/>

OMC [AAAI2022]

[Paper] [Training and Testing Tutorial] <br/> OMC introduces a double-check mechanism to make the "fake background" be tracked again. Specifically, we design a re-check network as the auxiliary to initial detections. If the target does not exist in the first-check predictions (i.e., the results of object detector), as a potential misclassified target, it has a chance to be restored by the re-check network, which searches targets through mining temporal cues. Note that, the re-check network innovatively expands the role of ID embedding from data association to motion forecasting by effectively propagating previous tracklets to the current frame with a small overhead. Even with multiple tracklets, our re-check network can still propagate with one forward pass by a simple matrix multiplication. Building on a strong baseline CSTrack, we construct a new one-shot tracker and achieve favorable gains.

<img src="https://github.com/JudasDie/SOTS/blob/MOT/demo/OMC.jpg" height="500" alt="OMC"/><br/>

AutoMatch [ICCV2021]

[Paper] [Raw Results] [Training and Testing Tutorial] [Demo] <br/> AutoMatch replaces the essence of Siamese tracking, i.e. the cross-correlation and its variants, to a learnable matching network. The underlying motivation is that heuristic matching network design relies heavily on expert experience. Moreover, we experimentally find that one sole matching operator is difficult to guarantee stable tracking in all challenging environments. In this work, we introduce six novel matching operators from the perspective of feature fusion instead of explicit similarity learning, namely Concatenation, Pointwise-Addition, Pairwise-Relation, FiLM, Simple-Transformer and Transductive-Guidance, to explore more feasibility on matching operator selection. The analyses reveal these operators' selective adaptability on different environment d