SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds

SimWorld is a simulation platform for developing and evaluating LLM/VLM AI agents in complex physical and social environments.

In summary, SimWorld supports three levels of usage:

Base: use the Base package (two lightweight city scenes + one empty map) for core agent interaction and quick testing.
Additional Environments: optionally expand Base with 100+ pre-built maps for richer scenarios.
Customization: bring your own UE environments, assets, and agent models to SimWorld for fully customized simulations.

See Setup and Make Your SimWorld for details.

📌 Table of Contents

🎬 Demonstration
🔥 News
💡 Introduction
🏗️ Architecture
🚀 Quick Tour — a minimum gym-like agent example
⚙️ Setup — Python / UE server install + quick start
📚 Configuration and API Reference — configs + commonly used APIs
🛠️ Make Your SimWorld — bring your own env/assets/agents
🔮 Next Steps
🤝 Contributing
⭐ Star History

🎬 Demonstration

<a href="https://www.youtube.com/watch?v=SfOifXTupgY" target="_blank" rel="noopener noreferrer"> <img src="docs/source/assets/video_cover.jpg" alt="SimWorld Demo Video" /> </a> <a href="https://www.youtube.com/@SimWorld-AI" target="_blank" rel="noopener noreferrer"> ▶ See all our demo videos on YouTube </a>

🔥 News

2026.1 SimWorld now supports importing customized environments and agents(doc)!
2025.11 The white paper of SimWorld is available on arxiv!
2025.9 SimWorld has been accepted to NeurIPS 2025 main track as a spotlight paper! 🎉
2025.6 The first formal release of SimWorld has been published! 🚀
2025.3 Our demo of SimWorld has been accepted by CVPR 2025 Demonstration Track! 🎉

💡 Introduction

SimWorld is built on Unreal Engine 5 and offers core capabilities to meet the needs of modern agent development. It provides:

Realistic, open-ended world simulation with accurate physics and language-based procedural generation.
Rich interface for LLM/VLM agents, supporting multi-modal perception and natural language actions.
Diverse and customizable physical and social reasoning scenarios, enabling systematic training and evaluation of complex agent behaviors like navigation, planning, and strategic cooperation.

🏗️ Architecture

SimWorld consists of three layers:

the Unreal Engine Backend, providing diverse and open-ended environments, rich assets and realistic physics simulation;
the Environment layer, supporting procedural city generation, language-driven scene editing, gym-like APIs for LLM/VLM agents and traffic simulation;
the Agent layer, enabling LLM/VLM agents to reason over multimodal observations and history while executing actions via a local action planner.

SimWorld's architecture is designed to be modular and flexible, supporting an array of functionalities such as dynamic world generation, agent control, and performance benchmarking. The components are seamlessly integrated to provide a robust platform for Embodied AI and Agents research and applications.

Project Structure

simworld/               # Python package
    local_planner/      # Local action planner component
    agent/              # Agent system
    assets_rp/          # Live editor component for retrieval and re-placing
    citygen/            # City layout procedural generator
    communicator/       # Core component to connect Unreal Engine
    config/             # Configuration loader and default config file
    llm/                # Basic llm class
    map/                # Basic map class and waypoint system
    traffic/            # Traffic system
    utils/              # Utility functions
    data/               # Default data files, e.g., object categories
    weather/            # Weather system
data/                   # Necessary input data
config/                 # Example configuration file and user configuration file
examples/                # Examples of usage, such as layout generation and traffic simulation
docs/                   # Documentation source files
README.md

🚀 Quick Tour

Here's a minimal example showing how to create an LLM-driven navigation task in SimWorld. This demo creates a humanoid agent that autonomously navigates to a target location using natural language reasoning and a Gym-like interface.

Note that the code is simplified for demonstration. For the complete implementation, see examples/gym_interface_demo.ipynb. More examples are available in the examples/ directory.

import sys
import os
import time
import re
import math
from simworld.config import Config
from simworld.communicator.communicator import Communicator
from simworld.communicator.unrealcv import UnrealCV
from simworld.llm.base_llm import BaseLLM
from simworld.map.map import Map
from simworld.agent.humanoid import Humanoid
from simworld.utils.vector import Vector

# 1. Define Agent class - uses LLM to decide navigation actions
class Agent():
    def __init__(self):
        self.llm = BaseLLM("gpt-4o")
        self.system_prompt = """ ... Your task is to ..."""

    def action(self, obs, target):
        position = obs['position']
        direction = obs['direction']

        # Calculate angle to target
        current_yaw = math.degrees(math.atan2(direction.y, direction.x))
        # ... calculate angle difference to target ...

        prompt = f""" ... Choose your next action ..."""

        action, _ = self.llm.generate_text(system_prompt=self.system_prompt, user_prompt=prompt)
        return action.strip()

# 2. Define Environment class - Gym-like interface
class Environment:
    def __init__(self, communicator, config=Config()):
        self.communicator = communicator
        self.agent = None
        self.target = None
        # ... initialize map and config ...

    def reset(self):
        """Spawn/reset humanoid and return initial observation."""
        spawn_location = Vector(0, 0)
        spawn_forward = Vector(1, 0)

        # ============ IMPORTANT: Spawn humanoid agent in UE world ============
        self.agent = Humanoid(  # <=========== Create humanoid agent
            communicator=self.communicator,
            position=spawn_location,
            direction=spawn_forward,
            config=self.config,
            map=self.map
        )
        self.communicator.spawn_agent(self.agent, name=None, model_path=agent_bp, type="humanoid")  # <=========== Spawn in UE
        self.target = Vector(1700, -1700)

        # ============ IMPORTANT: Get initial observation ============
        loc_3d = self.communicator.unrealcv.get_location(self.agent_name)  # <=========== Get agent position
        position = Vector(loc_3d[0], loc_3d[1])

        orientation = self.communicator.unrealcv.get_orientation(self.agent_name)  # <=========== Get agent orientation
        yaw = orientation[1]  # Yaw angle in degrees
        # Convert yaw to direction vector
        direction = Vector(math.cos(math.radians(yaw)), math.sin(math.radians(yaw)))

        ego_view = self.communicator.get_camera_observation(self.agent.camera_id, "lit")  # <=========== Get camera view

        observation = {
            'position': position,
            'direction': direction,  # unit vector
            'ego_view': ego_view     # RGB image
        }
        return observation

    def step(self, action):
        """Parse and execute action, return new observation and reward."""
        # ============ IMPORTANT: Execute humanoid movement commands ============
        if action.startswith("forward"):
            # Extract duration and execute forward movement
            self.communicator.humanoid_step_forward(self.agent.id, duration, direction=0)  # <=========== Move forward
        elif action.startswith("rotate"):
            # Extract angle and direction, execute rotation
            self.communicat

SimWorld

Install / Use

README