Segment and Track Anything (SAM-Track)

Online Demo: Technical Report:

Tutorial: tutorial-v1.6(audio),tutorial-v1.5 (Text), tutorial-v1.0 (Click & Brush)

<p align="center"> <img src="./assets/top.gif" width="880"> </p>

Segment and Track Anything is an open-source project that focuses on the segmentation and tracking of any objects in videos, utilizing both automatic and interactive methods. The primary algorithms utilized include the SAM (Segment Anything Models) for automatic/interactive key-frame segmentation and the DeAOT (Decoupling features in Associating Objects with Transformers) (NeurIPS2022) for efficient multi-object tracking and propagation. The SAM-Track pipeline enables dynamic and automatic detection and segmentation of new objects by SAM, while DeAOT is responsible for tracking all identified objects.

Impact

Segment and Track anything has been adopted as core infrastructure in published robotics research from MIT CSAIL and Harvard SEAS:

| Project | Institutions | Venue | Details | |---------|-------------|-------|---------| | Follow Anything (FAn) | MIT CSAIL · Harvard SEAS · Project CETI | IEEE RA-L | Open-vocabulary real-time drone tracking system. |

---

:loudspeaker:New Features

• [2024/4/23] We have added an audio-grounding feature that tracks the sound-making object within the video's soundtrack.

• [2023/5/12] We have authored a technical report for SAM-Track.

• [2023/5/7] We have added demo_instseg.ipynb, which uses Grounding-DINO to detect new objects in the key frames of a video. It can be applied in the fields of smart cities and autonomous driving.

• [2023/4/29] We have added advanced arguments for AOT-L: long_term_memory_gap and max_len_long_term.

  • long_term_memory_gap controls the frequency at which the AOT model adds new reference frames to its long-term memory. During mask propagation, AOT matches the current frame with the reference frames stored in the long-term memory.
  • Setting the gap value to a proper value helps to obtain better performance. To avoid memory explosion in long videos, we set a max_len_long_term value for the long-term memory storage, i.e. when the number of memory frames reaches the max_len_long_term value, the oldest memory frame will be discarded and a new frame will be added.

• [2023/4/26] Interactive WebUI 1.5-Version: We have added new features based on Interactive WebUI-1.0 Version.

  • We have added a new form of interactivity—text prompts—to SAMTrack.
  • From now on, multiple objects that need to be tracked can be interactively added.
  • Check out tutorial for Interactive WebUI 1.5-Version. More demos will be released in the next few days.

• [2023/4/26] Image-Sequence input: The WebUI now has a new feature that allows for input of image sequences, which can be used to test video segmentation datasets. Get started with the tutorial for Image-Sequence input.

• [2023/4/25] Online Demo: You can easily use SAMTrack in Colab for visual tracking tasks.

• [2023/4/23] Interactive WebUI: We have introduced a new WebUI that allows interactive user segmentation through strokes and clicks. Feel free to explore and have fun with the tutorial!

  • [2023/4/24] Tutorial V1.0: Check out our new video tutorials!
    • YouTube-Link: Tutorial for Interactively modify single-object mask for first frame of video、Tutorial for Interactively add object by click、Tutorial for Interactively add object by stroke.
    • Bilibili Video Link:Tutorial for Interactively modify single-object mask for first frame of video、Tutorial for Interactively add object by click、Tutorial for Interactively add object by stroke.
  • 1.0-Version is a developer version, please feel free to contact us if you encounter any bugs :bug:.

• [2023/4/17] SAMTrack: Automatically segment and track anything in video!

:fire:Demos

<div align=center>

</div>

This video showcases the segmentation and tracking capabilities of SAM-Track in various scenarios, such as street views, AR, cells, animations, aerial shots, and more.

:calendar:TODO

• [x] Colab notebook: Completed on April 25th, 2023.
• [x] 1.0-Version Interactive WebUI: Completed on April 23rd, 2023.
  • We will create a feature that enables users to interactively modify the mask for the initial video frame according to their needs. The interactive segmentation capabilities of Segment-and-Track-Anything is demonstrated in Demo8 and Demo9.
  • Bilibili Video Link: Demo8, Demo9.
• [x] 1.5-Version Interactive WebUI: Completed on April 26th, 2023.
  • We will develop a function that allows interactive modification of multi-object masks for the first frame of a video. This function will be based on Version 1.0. YouTube: Demo4, Demo5; Bilibili: Demo4, Demo5
  • Furthermore, we plan to include text prompts as an additional form of interaction. YouTube: Demo1, Demo2; Bilibili: Demo1, Demo2
• [ ] 2.x-Version Interactive WebUI
  • In version 2.x, the segmentation model will offer two options: SAM and SEEM.
  • We will develop a new function where the fixed-category object detection result can be displayed as a prompt.
  • We will enable SAM-Track to add and modify objects during tracking. YouTube: Demo6, Demo7; Bilibili: Demo6, Demo7

Demo1 showcases SAM-Track's ability to take the class of objects as prompt. The user gives the category text 'panda' to enable instance-level segmentation and tracking of all objects belonging to this category.

<div align=center>

</div>

Demo2 showcases SAM-Track's ability to take the text description as prompt. SAM-Track could segment and track target objects given the input that 'panda on the far left'.

<div align=center>

</div>

Demo3 showcases SAM-Track's ability to track numerous objects at the same time. SAM-Track is capable of automatically detecting newly appearing objects.

<div align=center>

</div>

Demo4 showcases SAM-Track's ability to take multiple modes of interactions as prompt. The user specified human and skateboard with click and brushstroke, respectively.

<div align=center>

</div>

Demo5 showcases SAM-Track's ability to refine the results of segment-everything. The user merges the tram as a whole with a single click.

<div align=center>

</div>

Demo6 showcases SAM-Track's ability to add new objects during tracking. The user annotates another car by rolling back to an intermediate frame.

<div align=center>