OpenWorldLib
Unified Codebase for Advanced World Models.
Install / Use
/learn @OpenDCAI/OpenWorldLibREADME
Welcome to join us open-source world model project ! <!-- omit in toc -->
<a href="https://github.com/OpenDCAI/OpenWorldLib"><img alt="Build" src="https://img.shields.io/github/stars/OpenDCAI/OpenWorldLib"></a> <!-- License --> <a href="https://github.com/OpenDCAI/OpenWorldLib/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/github/license/OpenDCAI/OpenWorldLib"></a> <!-- GitHub Issues --> <a href="https://github.com/OpenDCAI/OpenWorldLib/issues"><img alt="Issues" src="https://img.shields.io/github/issues/OpenDCAI/OpenWorldLib"></a> <a href="https://github.com/user-attachments/assets/35d48c4f-adb3-4f10-b30f-e7f4a245ab48"><img alt="Add me on WeChat" src="https://img.shields.io/badge/Connect_on-WeChat-07C160?style=flat-square&logo=wechat&logoColor=white"></a> <a href="https://arxiv.org/abs/2604.04707"><img alt="Paper" src="https://img.shields.io/badge/arXiv-2604.04707-b31b1b?logo=arxiv&logoColor=white"></a>
<!-- <img alt="Report" src="https://img.shields.io/badge/📄 Technical Report-Coming Soon-lightgrey"> -->Extension repo：[3D generation] | [VLA] | [simulator]
</div> <!-- add demo --> <div align="center"> <table> <tr> <td align="center"> <img src="https://github.com/user-attachments/assets/1c328d17-ef94-4d53-8dc2-f77ec3964a74" width="218" height="120"><br/> <b>Matrix-Game-2</b> </td> <td align="center"> <img src="https://github.com/user-attachments/assets/2fac45f5-d365-4794-bf38-45aee58f3d45" width="218" height="120"><br/> <b>Hunyuan-GameCraft</b> </td> <td align="center"> <img src="https://github.com/user-attachments/assets/c594ac11-1fc7-45da-ab2e-426e66e927f6" width="218" height="120"><br/> <b>Hunyuan-Worldplay</b> </td> </tr> <tr> <td align="center"> <img src="https://github.com/user-attachments/assets/e07456a3-869a-408c-9755-abe896f51c0e" width="218" height="120"><br/> <b>Lingbot-World</b> </td> <td align="center"> <img src="https://github.com/user-attachments/assets/ae38060b-f82f-48b1-89b5-6344e85e8354" width="218" height="120"><br/> <b>YUME-1.5</b> </td> <td align="center"> <img src="https://github.com/user-attachments/assets/c6b176fd-1ebe-4760-ab2f-6f2ba20445ca" width="218" height="120"><br/> <b>FlashWorld</b> </td> </tr> <tr> <td align="center"> <img src="https://github.com/user-attachments/assets/c4fa5245-e837-4e47-a4e4-585a61358f91" width="218" height="120"><br/> <b>Wan-2.2-IT2V</b> </td> <td align="center"> <img src="https://github.com/user-attachments/assets/a47b2e0e-b62c-45fb-bb99-6377f53e3f5e" width="218" height="120"><br/> <b>WoW</b> </td> <td align="center"> <img src="https://github.com/user-attachments/assets/c8ec425d-3815-4652-b2af-7e3c06e76b72" width="218" height="120"><br/> <b>Cosmos-Predict-2.5</b> </td> </tr> <tr> <td align="center"> <img src="https://github.com/user-attachments/assets/68c59c52-35c5-400c-a5a0-94c8eb802a79" width="218" height="120"><br/> <b>Pi3</b> </td> <td align="center"> <img src="https://github.com/user-attachments/assets/dab9c489-fcb8-412d-99af-9fdad4e76e0e" width="120" height="120"><br/> <b>Libero</b> </td> <td align="center"> <img src="https://github.com/user-attachments/assets/f8b4253b-9172-4afa-bd0e-ae0bc7ea8b25" width="120" height="120"><br/> <b>Ai2-THOR</b> </td> </tr> </table> </div>We define a world model as: A model or framework centered on perception, equipped with interaction and long-term memory capabilities, for understanding and predicting the complex world. Accordingly, 🎓 Multimodal Understanding, 🤖 Visual Action Prediction, and 🖼️ Visual Generation are all sub-tasks that a world model needs to accomplish.
We warmly welcome researchers to share their views on this framework or thoughts on world models in the Issues section. We also hope that you can submit valuable world-model-related methods to our framework via Pull Requests, or document and submit them to [awesome_world_models]. Feel free to give our repo a star 🌟 to follow the latest progress of OpenWorldLib!
Important Docs <!-- omit in toc -->
The following four documents are essential to this project (click to navigate):
- docs/planning.md: This document tracks the short-term optimization goals and future development plans for OpenWorldLib.
- docs/awesome_world_models.md: This document records cutting-edge research, related surveys, and open-source projects on world models.
- docs/installation.md: This document provides installation instructions for different methods in OpenWorldLib.
- Development Guide: This document provides a framework template for OpenWorldLib, intended to serve as a reference for code integration for developers in the world model community.
Table of Contents <!-- omit in toc -->
Features
Project Goals
The main goals of OpenWorldLib include:
- Establishing a unified and standardized world model framework to make the invocation of existing world-model-related code more consistent and well-structured;
- Integrating open-source world model research outcomes and systematically curating related papers for researchers' reference and use.
Supported Tasks
OpenWorldLib covers the following research directions related to World Models, We sincerely thank all the excellent methods included in this framework for their significant contributions to world model:
| Task Category | Sub-task | Representative Methods/Models | | :--- | :--- | :--- | | Video Generation | Navigation Generation | lingbot, matrix-game, hunyuan-worldplay, genie3, etc. | | | Long Video Generation | sora-2, veo-3, wan, etc. | | | Egocentric Interaction Generation | Hand2World, etc. | | 3D Scene Generation | 3D Scene Generation | flash-world, vggt, etc. | | Reasoning | VQA (Visual Question Answering) | spatialVLM, omnivinci and other VLMs with world understanding | | | VLA (Vision-Language-Action) | pi-0, pi-0.5, giga-brain, etc. |
Commonly used inference frameworks include: diffusers, DiffSynth, LightX2V
Getting Started
Installation
First, create a conda environment:
conda create -n "openworldlib" python=3.10 -y
conda activate "openworldlib"
Then install using the provided script:
cd OpenWorldLib
bash scripts/setup/default_install.sh
Some methods have special installation requirements. All installation scripts are located in ./scripts/setup.
📖 For the full installation guide, please refer to docs/installation.md
Quickstart
After installing the base environment, you can test matrix-game-2 generation and multi-turn interaction with the following commands:
cd OpenWorldLib
bash scripts/test_inference/test_nav_video_gen.sh matrix-game-2
bash scripts/test_stream/test_nav_video_gen.sh matrix-game-2
Scripts for other methods can be found under scripts/test_inference and scripts/test_stream. Currently, we are primarily using GPUs with 80GB and 141GB of VRAM for testing. In the future, we will test more models and provide updates in the ./docs/installation.md file.
Structure
To help developers and users better understand OpenWorldLib, we provide details about our codebase. The framework structure is as follows:
OpenWorldLib
├─ assets
├─ data # Test data
│ ├─ benchmarks
│ ├─ test_case
│ └─ ...
├─ docs # Documentation
├─ examples # Benchmark evaluation code
├─ scripts # All key test scripts
├─ src
│ └─ openworldlib # Main source path
│ ├─ base_models # Base models, provide basic modules for other parts
│ │ ├─ diffusion_model
│ │ ├─ llm_mllm_core
│ │ ├─ perception_core
│ │ └─ three_dimensions
│ ├─ memories # Memory module
│ │ ├─ reasoning
│ │ └─ visual_synthesis
│ ├─ operators # Input & interaction signal processing
│ ├─ pipelines # All runtime pipelines
│ ├─ reasoning # Reasoning module
│ │ ├─ audio_reasoning
│ │ ├─ general_reasoning
│ │ └─ spatial_reasoning
│ ├─ representations # Representation module
│ │ ├─ point_clouds_generation
│ │ └─ simulation_environment
│ └─ synthesis # Generation module
│ ├─ audio_generation
│ ├─ visual_generation
│ └─ vla_generation
├─ submodules # Auxiliary installs (e.g., diff-gaussian-raster)
├─ test # All test code
├─ test_stream # All interactive test code
└─ tools # Utilities
└─ vibe_code
When using OpenWorldLib, users typically call the pipeline class directly, which handles weight loading, environment initialization
Related Skills
qqbot-channel
352.2kQQ 频道管理技能。查询频道列表、子频道、成员、发帖、公告、日程等操作。使用 qqbot_channel_api 工具代理 QQ 开放平台 HTTP 接口，自动处理 Token 鉴权。当用户需要查看频道、管理子频道、查询成员、发布帖子/公告/日程时使用。
docs-writer
100.6k`docs-writer` skill instructions As an expert technical writer and editor for the Gemini CLI project, you produce accurate, clear, and consistent documentation. When asked to write, edit, or revie
model-usage
352.2kUse CodexBar CLI local cost usage to summarize per-model usage for Codex or Claude, including the current (most recent) model or a full model breakdown. Trigger when asked for model-level usage/cost data from codexbar, or when you need a scriptable per-model summary from codexbar cost JSON.
arscontexta
3.1kClaude Code plugin that generates individualized knowledge systems from conversation. You describe how you think and work, have a conversation and get a complete second brain as markdown files you own.
