<img src="docs/FreeAskWorldLOGO.png" alt="FreeAskWorld Logo" width="320"> <h1 align="center">FreeAskWorld Simulator (AAAI26 Oral)</h1> An Interactive and Closed-Loop Simulator for Human-Centric Embodied AI   <a href="https://arxiv.org/abs/2511.13524" target="_blank"> <img src="https://img.shields.io/badge/arXiv-2511.13524-b31b1b.svg" alt="arXiv"> </a> <a href="https://huggingface.co/datasets/Astronaut-PENG/FreeAskWorld" target="_blank"> <img src="https://img.shields.io/badge/HuggingFace-FreeAskWorld-yellow.svg" alt="HuggingFace Dataset"> </a>  <a href="LICENSE" target="_blank"> <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="Apache License"> </a>  <a href="https://github.com/doraemonaaaa/FreeAskWorldConnector" target="_blank"> <img src="https://img.shields.io/badge/Baseline-Closed--Loop%20Framework%20-0a7cff.svg" alt="Closed-Loop Framework"> </a> <a href="https://github.com/AIR-DISCOVER/FreeAD" target="_blank"> <img src="https://img.shields.io/badge/Related%20Work-FreeAD-blueviolet.svg" alt="FreeAD Project"> </a> <a href="https://github.com/doraemonaaaa/FreeAskAgent" target="_blank"> <img src="https://img.shields.io/badge/Related%20Work-FreeAskAgent-blueviolet.svg" alt="FreeAskAgent Project"> </a> FreeAskWorld is an interactive simulation framework that integrates large language models (LLMs) for high-level planning and socially grounded interaction in embodied AI. <img src="docs/SystemOverview.jpg" alt="System Overview" width="100%"> <img src="docs/PeopleSimulationFramework.jpg" alt="People Simulation Framework" width="100%"> <img src="docs/Style.jpg" alt="FreeAskWorld Homepage" width="100%">

Project Milestones

[x] 📝 Paper Publication: Published the main research paper describing FreeAskWorld.
[x] 📊 Data Processing Code Release: Released code for preprocessing, data cleaning, and annotation pipelines.
[x] 🎥 Presentation Video: Released project presentation video.
[ ] 🛠️ Simulator Code Release: Publish the core simulation code for developers and external collaborators.
[x] 🤖 Agent Robot Integration: Integrate agent interfaces (OpenClaw / Codex / Claude / custom) to access and interact with robots inside the FreeAskWorld simulation environment.
[ ] 📚 Usage Tutorial: Create a comprehensive tutorial for using the FreeAskWorld simulator, including setup, configuration, and example workflows.
[ ] 🧑‍💻 API Documentation: Provide thorough documentation of the simulator’s API for seamless integration and extension.
[ ] 🎮 Steam Release: Prepare and publish the FreeAskWorld simulator on Steam for broader accessibility.

Agent integration note: the current recommended local path is ROS2-first, because the local Unity simulator is configured in ROS2 mode on 127.0.0.1:10000. See docs/agent_ros2_integration.md. The ROS2 path now includes a live-capable rclpy transport when external ROS2 runtime dependencies are present, while still preserving scaffold-only fallback behavior. For live ROS2 usage, a repo-local .ros2_venv or equivalent ROS-compatible Python environment may be needed to avoid ROS Humble rclpy ABI mismatch, and the wrapper at scripts/agent_ros2_cli.sh now auto-activates .ros2_venv when present. The additive closed_loop websocket Agent bridge remains available, but for this Unity setup it should be treated as experimental and future-facing rather than the primary runtime path.

Agent config entry paths (for OpenClaw / Claude Code / Codex / custom agent adapters):

ROS2 integration doc: docs/agent_ros2_integration.md
ROS2 package: integrations/agent_ros2/
ROS2 wrapper CLI: scripts/agent_ros2_cli.sh
Legacy closed-loop agent bridge prototypes are archived under archived/closed_loop-agent/ and are not part of the current recommended control path.

Help auto install Agent ROS2 Integration. Other AI agents can discover these paths and install/configure automatically.

Canonical entry command for auto-discovery:

bash scripts/agent_ros2_cli.sh --help

First-time setup notes (read this before install)

FreeAskWorld now includes a repo-owned local runtime path for ROS2-first live testing.

Requirements

Python 3.10
ROS2 Humble installed manually
ROS2 manual setup guide: docs/ros2_setup.md

Option 1 — Agent install (recommended)

In an agent session, use this exact instruction:

Install all envs by scripts/setup_envs.sh

Option 2 — User command line install

From the repo root, run:

cd ~/research/FreeAskWorld && bash scripts/setup_envs.sh

What scripts/setup_envs.sh does:

checks whether a repo-local environment already exists
reuses it if present instead of blindly creating a new one
otherwise creates .ros2_venv with Python 3.10
installs the minimal Python packages needed for live testing
warns clearly if ROS2 Humble is not installed yet
points you to the manual ROS2 setup guide if system ROS2 is missing

After setup:

source .ros2_venv/bin/activate
scripts/start_local_runtime.sh
scripts/status_local_runtime.sh
curl http://127.0.0.1:8787/healthz
STEP_SECONDS=2 OBSERVE_SECONDS=1 scripts/run_live_smoke.sh
scripts/stop_local_runtime.sh

run_live_smoke.sh now visibly executes all major player actions for a few seconds each and prints step-by-step results plus observation summaries.

Shortest interactive player-control examples:

scripts/player_cmd.sh status
scripts/player_cmd.sh observe 1
scripts/player_cmd.sh forward 0.5
scripts/player_cmd.sh left 30
scripts/player_cmd.sh right 30
scripts/player_cmd.sh around
scripts/player_cmd.sh stop
scripts/player_cmd.sh wait 1
scripts/player_cmd.sh ask "Where is the target?"
scripts/player_cmd.sh action '{"action":"move_forward","parameters":{"distance_m":0.5}}'

Expected behavior for the minimal checks above:

--help prints CLI usage.
status --output-json runs even without Unity connected.
In scaffold-only mode, transport_ready: false does not by itself mean the repo is broken.

For live validation, use:

scripts/start_local_runtime.sh
STEP_SECONDS=2 OBSERVE_SECONDS=1 scripts/run_live_smoke.sh

This visibly runs the main player actions (forward, left, right, around, wait, ask, stop), prints each step result, captures observations between steps, and writes a JSON report.

If --ros2-live fails immediately on a fresh machine, check these first:

ROS2 Humble is installed manually; use docs/ros2_setup.md.
.ros2_venv exists and includes at least numpy plus the local runtime Python deps.
The Unity/ROS2 backend is actually running and reachable.
The machine allows DDS/UDP/shared-memory transport required by ROS2 middleware.
The ROS log directory is writable (for example, set ROS_LOG_DIR=/tmp/roslog if needed).

🎥 Demos

Simulator Presentation Demonstrates the main functions of this simulator.

<video width="800" controls> <source src="docs/Presentation.mp4" type="video/mp4"> 你的浏览器不支持 HTML5 视频播放，请点击下方链接下载。 </video> <a href="docs/Presentation.mp4">📥 Download Simulator Presentation Video</a>

Simulator APP Presentation Demonstrates the main functions of this simulator.

<video width="800" controls> <source src="docs/APP Presentation.mp4" type="video/mp4"> 你的浏览器不支持 HTML5 视频播放，请点击下方链接下载。 </video> <a href="docs/APP Presentation.mp4">📥 Download APP Presentation Video</a>

ROS2 Example Demonstrates the ROS2 RGBD SLAM in our simulator.

<video width="800" controls> <source src="docs/RGBD SLAM Presentation.mp4" type="video/mp4"> 你的浏览器不支持 HTML5 视频播放，请点击下方链接下载。 </video> <a href="docs/RGBD SLAM Presentation.mp4">📥 Download ROS2 Example Video</a>

📌 Introduction

As embodied intelligence progresses, simulation platforms must evolve beyond low-level physics toward human-centric, socially interactive environments.
FreeAskWorld introduces:

A closed-loop interactive simulator
A scalable human-agent world modeling framework
A modular data generation pipeline
A new benchmark: Direction Inquiry Task, extending VLN to active question-asking & guidance following

This repo contains simulator code and baseline models from our AAAI 2026 paper.

✨ Key Features

| Feature | Description | |---|---| | 🤖 LLM-Powered Agents | Intention modeling, reasoning, natural dialog, instruction generation | | 🚶 Realistic Humans | Personalized profiles, schedules, motion & navigation styles | | 🌦️ Dynamic World | Weather, lighting, traffic, and scene randomization | | 🔁 Closed-Loop Sync | WebSocket-based state exchange for real-time model interaction | | 🧩 Direction Inquiry Task | Agents ask for help, interpret human guidance, adapt plans | | 📦 Large-Scale Data | 6 tasks · 16 object categories · 63,429 frames · 17+ hours | | 🔄 Data Generation Pipeline | Modular pipeline for generating embodied ai data |

Synthetic Data Generation

docs/OccupancyMapGenerationContrast.jpg docs/SyhteticDataPic.jpg

FreeAskWorld

Install / Use

README