DragAnything

<div align="center"> DragAnything: Motion Control for Anything using Entity Representation <div>

<div align="center"> <a href="https://weijiawu.github.io/draganything_page/"><img src="https://img.shields.io/static/v1?label=Project%20Page&message=Github&color=blue&logo=github-pages"></a> &ensp; <a href="https://arxiv.org/abs/2403.07420/"><img src="https://img.shields.io/static/v1?label=Paper&message=Arxiv&color=red&logo=arxiv"></a> &ensp; </div> <p align="center"> <img src="./assets/1709656085862.jpg" width="800px"/> <br> </p>

:notes: Updates

<!--- [ ] Mar. 13, 2024. Release the train code in **three month**.-->

• [x] July. 1, 2024. DragAnything has been accepted by ECCV 2024!
• [x] Mar. 24, 2024. Support interactive demo with gradio.
• [x] Mar. 13, 2024. Release the inference code.
• [x] Mar. 12, 2024. Rep initialization.

---

🐱 Abstract

We introduce DragAnything, which utilizes an entity representation to achieve motion control for any object in controllable video generation. Comparison to existing motion control methods, DragAnything offers several advantages. Firstly, trajectory-based is more user-friendly for interaction, when acquiring other guidance signals (\eg{} masks, depth maps) is labor-intensive. Users only need to draw a line~(trajectory) during interaction. Secondly, our entity representation serves as an open-domain embedding capable of representing any object, enabling the control of motion for diverse entities, including background. Lastly, our entity representation allows simultaneous and distinct motion control for multiple objects. Extensive experiments demonstrate that our DragAnything achieves state-of-the-art performance for FVD, FID, and User Study, particularly in terms of object motion control, where our method surpasses the previous state of the art (DragNUWA) by 26% in human voting.

---

User-Trajectory Interaction with SAM

<table class="center"> <tr> <td style="text-align:center;"><b>Input Image</b></td> <td style="text-align:center;"><b>Drag point with SAM</b></td> <td style="text-align:center;"><b>2D Gaussian Trajectory</b></td> <td style="text-align:center;"><b>Generated Video</b></td> </tr> <tr> <td><img src="./assets/1709660422197.jpg" width="177" height="100"></td> <td><img src="./assets/1709660459944.jpg" width="177" height="100"></td> <td><img src="./assets/image28 (3).gif" width="177" height="100"></td> <td><img src="./assets/image28 (2).gif" width="177" height="100"></td> </tr> <tr> <td><img src="./assets/1709660422197.jpg" width="177" height="100"></td> <td><img src="./assets/1709660471568.jpg" width="177" height="100"></td> <td><img src="./assets/image2711.gif" width="177" height="100"></td> <td><img src="./assets/image27 (1)1.gif" width="177" height="100"></td> </tr> <tr> <td><img src="./assets/1709660422197.jpg" width="177" height="100"></td> <td><img src="./assets/1709660965701.jpg" width="177" height="100"></td> <td><img src="./assets/image29111.gif" width="177" height="100"></td> <td><img src="./assets/image29 (1)1.gif" width="177" height="100"></td> </tr> <tr> <td><img src="./assets/1709660422197.jpg" width="177" height="100"></td> <td><img src="./assets/1709661150250.jpg" width="177" height="100"></td> <td><img src="./assets/image30 (1)1.gif" width="177" height="100"></td> <td><img src="./assets/image3011.gif" width="177" height="100"></td> </tr> </table>

Comparison with DragNUWA

<table class="center"> <tr> <td style="text-align:center;"><b>Model</b></td> <td style="text-align:center;"><b>Input Image and Drag</b></td> <td style="text-align:center;"><b>Generated Video</b></td> <td style="text-align:center;"><b>Visualization for Pixel Motion</b></td> </tr> <tr> <td style="text-align:center;"><b>DragNUWA</b></td> <td><img src="./assets/1709661872632.jpg" width="177" height="100"></td> <td><img src="./assets/image63111.gif" width="177" height="100"></td> <td><img src="./assets/image6411.gif" width="177" height="100"></td> </tr> <tr> <td style="text-align:center;"><b>Ours</b></td> <td><img src="./assets/1709662077471.jpg" width="177" height="100"></td> <td><img src="./assets/image65111.gif" width="177" height="100"></td> <td><img src="./assets/image6611.gif" width="177" height="100"></td> </tr> <tr> <td style="text-align:center;"><b>DragNUWA</b></td> <td><img src="./assets/1709662293661.jpg" width="177" height="100"></td> <td><img src="./assets/image77.gif" width="177" height="100"></td> <td><img src="./assets/image76.gif" width="177" height="100"></td> </tr> <tr> <td style="text-align:center;"><b>Ours</b></td> <td><img src="./assets/1709662429867.jpg" width="177" height="100"></td> <td><img src="./assets/image75.gif" width="177" height="100"></td> <td><img src="./assets/image74.gif" width="177" height="100"></td> </tr> <tr> <td style="text-align:center;"><b>DragNUWA</b></td> <td><img src="./assets/1709662596207.jpg" width="177" height="100"></td> <td><img src="./assets/image84.gif" width="177" height="100"></td> <td><img src="./assets/image85.gif" width="177" height="100"></td> </tr> <tr> <td style="text-align:center;"><b>Ours</b></td> <td><img src="./assets/1709662724643.jpg" width="177" height="100"></td> <td><img src="./assets/image87.gif" width="177" height="100"></td> <td><img src="./assets/image88.gif" width="177" height="100"></td> </tr> </table>

More Demo

<table class="center"> <tr> <td style="text-align:center;"><b>Drag point with SAM</b></td> <td style="text-align:center;"><b>2D Gaussian</b></td> <td style="text-align:center;"><b>Generated Video</b></td> <td style="text-align:center;"><b>Visualization for Pixel Motion</b></td> </tr> <tr> <td><img src="./assets/1709656550343.jpg" width="177" height="100"></td> <td><img src="./assets/image188.gif" width="177" height="100"></td> <td><img src="./assets/image190.gif" width="177" height="100"></td> <td><img src="./assets/image189.gif" width="177" height="100"></td> </tr> <tr> <td><img src="./assets/1709657635807.jpg" width="177" height="100"></td> <td><img src="./assets/image187 (1).gif" width="177" height="100"></td> <td><img src="./assets/image186.gif" width="177" height="100"></td> <td><img src="./assets/image185.gif" width="177" height="100"></td> </tr> <tr> <td><img src="./assets/1709658516913.jpg" width="177" height="100"></td> <td><img src="./assets/image158.gif" width="177" height="100"></td> <td><img src="./assets/image159.gif" width="177" height="100"></td> <td><img src="./assets/image160.gif" width="177" height="100"></td> </tr> <tr> <td><img src="./assets/1709658781935.jpg" width="177" height="100"></td> <td><img src="./assets/image163.gif" width="177" height="100"></td> <td><img src="./assets/image161.gif" width="177" height="100"></td> <td><img src="./assets/image162.gif" width="177" height="100"></td> </tr> <tr> <td><img src="./assets/1709659276722.jpg" width="177" height="100"></td> <td><img src="./assets/image165.gif" width="177" height="100"></td> <td><img src="./assets/image167.gif" width="177" height="100"></td> <td><img src="./assets/image166.gif" width="177" height="100"></td> </tr> <tr> <td><img src="./assets/1709659787625.jpg" width="177" height="100"></td> <td><img src="./assets/image172.gif" width="177" height="100"></td> <td><img src="./assets/Our_Motorbike_cloud_floor.gif" width="177" height="100"></td> <td><img src="./assets/image171.gif" width="177" height="100"></td> </tr> </table>

Various Motion Control

<table class="center"> <tr> <td style="text-align:center;"><b>Drag point with SAM</b></td> <td style="text-align:center;"><b>2D Gaussian</b></td> <td style="text-align:center;"><b>Generated Video</b></td> <td style="text-align:center;"><b>Visualization for Pixel Motion</b></td> </tr> <tr> <td><img src="./assets/1709663429471.jpg" width="177" height="100"></td> <td><img src="./assets/image265.gif" width="177" height="100"></td> <td><img src="./assets/image265 (1).gif" width="177" height="100"></td> <td><img src="./assets/image268.gif" width="177" height="100"></td> </tr> <tr> <td><img src="./assets/1709663831581.jpg" width="177" height="100"></td> <td><img src="./assets/image274.gif" width="177" height="100"></td> <td><img src="./assets/image274 (1).gif" width="177" height="100"></td> <td><img src="./assets/image276.gif" width="177" height="100"></td> </tr> <tr> <td style="text-align:center;" colspan="4"><b>(a) Motion Control for Foreground</b></td> </tr> <tr> <td><img src="./assets/1709664593048.jpg" width="177" height="100"></td> <td><img src="./assets/image270.gif" width="177" height="100"></td> <td><img src="./assets/image270 (1).gif" width="177" height="100"></td> <td><img src="./assets/image269.gif" width="177" height="100"></td> </tr> <tr> <td><img src="./assets/1709664834397.jpg" width="177" height="100"></td> <td><img src="./assets/image271.gif" width="177" height="100"></td> <td><img src="./assets/image271 (1).gif" width="177" height="100"></td> <td><img src="./assets/image272.gif" width="177" height="100"></td> </tr> <tr> <td style="text-align:center;" colspan="4"><b>(b) Motion Control for Background</b></td> </tr> <tr> <td><img src="./assets/1709665073460.jpg" width="177" height="100"></td> <td><img src="./assets/image279.gif" width="177" height="100"></td> <td><img src="./assets/image278.gif" width="177" height="100"></td> <td><img src="./assets/image277.gif" width="177" height="100"></td> </tr> <tr> <td><img src="./assets/1709665252573.jpg" width="177" height="100"></td>