Learning Adaptable World Models with Latent Actions

Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, Chuang Gan

<hr style="border: 2px solid gray;"></hr>

TL;DR: AdaWorld is a highly adaptable world model pretrained with continuous latent actions from thousands of environments, enabling zero-shot action transfer, fast adaptation, and new skill acquisition with minimal finetuning.

We introduce latent actions as a unified condition for action-aware pretraining from videos. AdaWorld can readily transfer actions across contexts without training. By initializing the control interface with the corresponding latent actions, AdaWorld can also be adapted into specialized world models efficiently and achieve significantly better planning results.

<hr style="border: 2px solid gray;"></hr>

• Action transfer (source video → target scene)

<div id="top" align="center"> <p align="center"> <img src="assets/transfer.gif" width="1000px" > </p> </div>

• Visual planning (action-agnostic vs. AdaWorld)

<div id="top" align="center"> <p align="center"> <img src="assets/planning.gif" width="1000px" > </p> </div> <hr style="border: 2px solid gray;"></hr>

🕹️ Getting Started

• Installation
• Training
• Action Transfer
• World Model Adaptation
• Visual Planning
• Trouble Shooting

❣️ Acknowledgement

Our idea is implemented based on Vista and Jafar. Thanks for their great open-source work!

🌟 Citation

If any parts of our paper and code help your research, please consider citing us and giving a star to our repository.

@inproceedings{gao2025adaworld,
 title={AdaWorld: Learning Adaptable World Models with Latent Actions},
 author={Gao, Shenyuan and Zhou, Siyuan and Du, Yilun and Zhang, Jun and Gan, Chuang},
 booktitle={International Conference on Machine Learning (ICML)},
 year={2025}
}

✉️ Contact

If you have any questions or comments, feel free to contact me through email (sygao@connect.ust.hk). Suggestions and collaborations are also highly welcome!