Segment Anything Meets Point Tracking

Segment Anything Meets Point Tracking
Frano Rajič, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, Fisher Yu
ETH Zürich, HKUST, EPFL

We propose SAM-PT, an extension of the Segment Anything Model (SAM) for zero-shot video segmentation. Our work offers a simple yet effective point-based perspective in video object segmentation research. For more details, refer to our paper.

Video Object Segmentation Demo

Annotators only provide a few points to denote the target object at the first video frame to get video segmentation results. Please visit our project page for more visualizations, including qualitative results on DAVIS 2017 videos and more Avatar clips.

<p float="left"> <img alt="street" src="assets/street.gif?raw=true" width="48%" /> <img alt="bees" src="assets/bees.gif?raw=true" width="48%" /> <img alt="avatar" src="assets/avatar.gif?raw=true" width="48%" /> <img alt="horsejump-high" src="assets/horsejump-high.gif?raw=true" width="48%" /> </p>

Interactive Point-Based Video Segmentation

Annotators can interactively add or remove points to refine the segmentation results.

<p float="left"> <img alt="camel" src="assets/interactive-camel.gif?raw=true" width="96.5%" /> <img alt="drift" src="assets/interactive-drift-straight.gif?raw=true" width="96.5%" /> <img alt="loading" src="assets/interactive-loading.gif?raw=true" width="96.5%" /> </p>

Documentation

Explore our step-by-step guides to get up and running:

1. Getting Started: Learn how to set up your environment and run the demo.
2. Prepare Datasets: Instructions on acquiring and prepping necessary datasets.
3. Prepare Checkpoints: Steps to fetch model checkpoints.
4. Running Experiments: Details on how to execute experiments.

Acknowledgments

We want to thank SAM, PIPS, CoTracker, HQ-SAM, MobileSAM, XMem, and Mask2Former for publicly releasing their code and pretrained models.

Citation

If you find SAM-PT useful in your research or if you refer to the results mentioned in our work, please star :star: this repository and consider citing :pencil::

@article{sam-pt,
  title   = {Segment Anything Meets Point Tracking},
  author  = {Rajič, Frano and Ke, Lei and Tai, Yu-Wing and Tang, Chi-Keung and Danelljan, Martin and Yu, Fisher},
  journal = {arXiv:2307.01197},
  year    = {2023}
}