WorldLens

<p align="right">English | <a href="./README_CN.md">简体中文</a></p> <p align="center"> <img src="docs/figures/worldbench.gif" width="12.8%" align="center"> <h1 align="center"> <strong>WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World</strong> </h1> <p align="center"> <strong>:earth_asia: WorldBench Team</strong> </p> <p align="center"> <a href="https://worldbench.github.io/assets_common/papers/worldlens.pdf" target='_blank'> <img src="https://img.shields.io/badge/Paper-%F0%9F%93%96-darkred"> </a>&nbsp; <a href="https://worldbench.github.io/worldlens" target='_blank'> <img src="https://img.shields.io/badge/Project-%F0%9F%94%97-blue"> </a>&nbsp; <a href="https://worldbench.github.io/worldlens" target='_blank'> <img src="https://img.shields.io/badge/Leaderboard-%F0%9F%94%97-yellow"> </a>&nbsp; <a href="https://huggingface.co/datasets/worldbench/videogen" target='_blank'> <img src="https://img.shields.io/badge/Dataset-%F0%9F%94%97-green"> </a>&nbsp; <a href="" target='_blank'> <img src="https://visitor-badge.laobi.icu/badge?page_id=worldbench.WorldLens"> </a> </p> </p>

| <img src="docs/figures/teaser.png" alt="teaser" width="100%"> | | :-: |

:grey_question: Is your driving world model an all-around player?

• This work presents WorldLens, a unified benchmark encompassing evaluations on $^1$Generation, $^2$Reconstruction, $^3$Action-Following, $^4$Downstream Task, and $^5$Human Preference, across a total of 24 dimensions spanning visual realism, geometric consistency, functional reliability, and perceptual alignment.
• We observe no single model dominates across all axes, highlighting the need for balanced progress toward physically and behaviorally realistic world modeling.
• For additional visual examples, kindly refer to our :earth_asia: Project Page.

:books: Citation

If you find this work helpful for your research, please kindly consider citing our papers:

@article{worldlens,
    title   = {{WorldLens}: Full-Spectrum Evaluations of Driving World Models in Real World},
    author  = {Ao Liang and Lingdong Kong and Tianyi Yan and Hongsi Liu and Wesley Yang and Ziqi Huang and Wei Yin and Jialong Zuo and Yixuan Hu and Dekai Zhu and Dongyue Lu and Youquan Liu and Guangfeng Jiang and Linfeng Li and Xiangtai Li and Long Zhuo and Lai Xing Ng and Benoit R. Cottereau and Changxin Gao and Liang Pan and Wei Tsang Ooi and Ziwei Liu},
    journal = {arXiv preprint arXiv:2512.10958},
    year    = {2025}
}

@article{survey_3d_4d_world_models,
    title   = {{3D} and {4D} World Modeling: A Survey},
    author  = {Lingdong Kong and Wesley Yang and Jianbiao Mei and Youquan Liu and Ao Liang and Dekai Zhu and Dongyue Lu and Wei Yin and Xiaotao Hu and Mingkai Jia and Junyuan Deng and Kaiwen Zhang and Yang Wu and Tianyi Yan and Shenyuan Gao and Song Wang and Linfeng Li and Liang Pan and Yong Liu and Jianke Zhu and Wei Tsang Ooi and Steven C. H. Hoi and Ziwei Liu},
    journal = {arXiv preprint arXiv:2509.07996},
    year    = {2025}
}

Updates

• [12/2025] - The official :balance_scale: WorldLens Leaderboard is online at HuggingFace Spaces. We invite researchers and practitioners to submit their models for evaluation on the leaderboard, enabling consistent comparison and supporting progress in world model research.
• [12/2025] - A collection of 3D and 4D world models is avaliable at :hugs: awesome-3d-4d-world-models.
• [12/2025] - The Project Page is online. :rocket:

Outline

• WorldLens Benchmark
• WorldLens Leaderboard
• Installation
• Data Preparation
• Getting Started
• WorldLens-26K
• WorldLens-Agent
• TODO List
• License
• Acknowledgements
• Related Projects

:earth_asia: WorldLens Benchmark

| <img src="docs/figures/bench.png" alt="framework" width="100%">| | :-: |

• Generative world models must go beyond visual realism to achieve geometric consistency, physical plausibility, and functional reliability. WorldLens is a unified benchmark that evaluates these capabilities across five complementary aspects - from low-level appearance fidelity to high-level behavioral realism.

• Each aspect is decomposed into fine-grained, interpretable dimensions, forming a comprehensive framework that bridges human perception, physical reasoning, and downstream utility.

For additional details and visual examples, kindly refer to our :books: Paper and :earth_asia: Project Page.

:balance_scale: WorldLens Leaderboard

|||| |:-:|:-:|:-| | <img src="docs/icons/generation.gif" width="100"> | Generation | Measuring whether a model can synthesize visually realistic, temporally stable, and semantically consistent scenes. Even state-of-the-art models that achieve low perceptual error (e.g., LPIPS, FVD) often suffer from view flickering or motion instability, revealing the limits of current diffusion-based architectures. | <img src="docs/icons/reconstruction.gif" width="100"> | Reconstruction | Probing whether generated videos can be reprojected into a coherent 4D scene using differentiable rendering. Models that appear sharp in 2D frequently collapse when reconstructed, producing geometric "floaters": a gap that exposes how temporal coherence remains weakly coupled in most pipelines. | <img src="docs/icons/action-following.gif" width="100"> | Action-Following | Testing if a pre-trained action planner can operate safely inside the generated world. High open-loop realism does not guarantee safe closed-loop control; almost all existing world models trigger collisions or off-road drifts, underscoring that photometric realism alone cannot yield functional fidelity. | <img src="docs/icons/downstream.gif" width="100"> | Downstream Task | Evaluating whether the synthetic data support downstream perception models trained on real-world datasets. Even visually appealing worlds may degrade detection or segmentation accuracy by 30-50%, highlighting that alignment to task distributions, not just image quality, is vital for practical usability. | <img src="docs/icons/human-preference.gif" width="100"> | Human Preference | Capturing subjective scores such as world realism, physical plausibility, and behavioral safety through large-scale human annotations. Our study reveals that models with strong geometric consistency are generally rated as more "real", confirming that perceptual fidelity is inseparable from structural coherence. ||||

Leaderboard

An interactive :balance_scale: WorldLens Leaderboard is online at :hugs: HuggingFace Spaces. We invite researchers and practitioners to submit their models for evaluation on the leaderboard, enabling consistent comparison and supporting progress in world model research.

<details open> <summary>&nbsp<b>Benchmarked Models</b></summary>

• [x] MagicDrive, ICLR 2023.
• [x] Panacea, CVPR 2024.
• [x] DreamForge, arXiv 2024.
• [x] DriveDreamer-2, AAAI 2025.
• [x] DrivingSphere, CVPR 2025.
• [x] OpenDWM, CVPR 2025.
• [x] MagicDrive-V2, ICCV 2025.
• [x] DiST-4D, ICCV 2025.
• [x] RLGF, NeurIPS 2025.
• [x] X-Scene, NeurIPS 2025.
• [ ] . . .

</details>

:gear: Installation

The WorldLens evaluation toolkit is developed and tested under Python 3.9 + CUDA 11.8. We recommend using Conda to manage the environment.

• Create Environment:

conda create -n worldbench python=3.9.20
conda activate worldbench

• Install PyTorch:

pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 \
    --index-url https://download.pytorch.org/whl/cu118

• Install MMCV (with CUDA):

cd worldbench/third_party/mmcv-1.6.0
MMCV_WITH_OPS=1 pip install -e .

Note: We modified the C++ standard to C++17 for better compatibility. You may adjust it in worldbench/third_party/mmcv-1.6.0/setup.py based on your system.

• Install MMSegmentation:

pip install https://github.com/open-mmlab/mmsegmentation/archive/refs/tags/v0.30.0.zip

• Install MMDetection:

pip install mmdet==2.28.2

• Install BEVFusion-based MMDet3D:

git clone --recursive https://github.com/worldbench/WorldLens.git
cd worldbench/third_party/bevfusion
python setup.py develop

Additional Notes:

1. C++ standard was updated to C++17.
2. We modified the sparse convolution import logic at worldbench/third_party/bevfusion/mmdet3d/ops/spconv/conv.py.

• Install MMDetection3D (v1.0.0rc6):

cd worldbench/third_party/mmdetection3d-1.0.0rc6
pip install -v -e .

Required dependency versions:

numpy == 1.23.5
numba == 0.53.0

• Pretrained Models WorldLens relies on several pretrained models (e.g., CLIP, segmentation, depth networks). Please download them from HuggingFace and place them under: ./pretrained_models/

:hotsprings: Data Preparation

Here we take nuScenes as an example. Required Files:

• nuScenes official dataset
• 12 Hz interpolated annotations from ECCV 2024 Workshop – CODA Track 2
• Tracking & temporal .pkl files from [HuggingFace – WorldLens Data Preparation](https://huggingface.co/datas