🌍 Learn to Drive in "Dreams": CarDreamer 🚗

<div align="center"> <a href="https://huggingface.co/ucd-dare/CarDreamer/tree/main"> <img src="https://img.icons8.com/?size=32&id=sop9ROXku5bb" alt="HuggingFace Checkpoints" /> HuggingFace Checkpoints </a>  |  <a href="https://car-dreamer.readthedocs.io/en/latest/"> <img src="https://img.icons8.com/nolan/32/api.png" alt="CarDreamer API Documents" /> CarDreamer API Documents </a>  |  <a href="https://ieeexplore.ieee.org/document/10714437"> <img src="https://img.icons8.com/?size=32&id=48326&format=png" alt="IEEE IoT" /> Paper </a>  |  <a href="https://ucd-dare.github.io/cardreamer.github.io/"> <img src="https://img.icons8.com/?size=32&id=X-WB1cntO5xU&format=png&color=000000" alt="Project Page" /> Project Page </a> </div>

Unleash the power of imagination and generalization of world models for self-driving cars.

[!NOTE]

October 2024: CarDreamer has been accepted by IEEE IoT!

August 2024: Support transmission error in intention sharing.

August 2024: Created a right turn random task.

July 2024: Created a stop-sign task and a traffic-light task.

July 2024: Uploaded all the task checkpoints to HuggingFace

May 2024 Released arXiv preprint.

Can world model based reinforcement learning train autonomous driving agents via imagination of traffic dynamics? The answer is YES!

Integrating the high-fidelity CARLA simulator with world models, we are able to train a world model that not only learns complex environment dynamics but also have an agent interact with the neural network "simulator" to learn to drive.

Simply put, in CarDreamer the agent can learn to drive in a "dream world" from scratch, mastering maneuvers like overtaking and right turns, and avoiding collisions in heavy traffic—all within an imagined world!

CarDreamer offers customizable observability, multi-modal observation spaces, and intention-sharing capabilities. Our paper presents a systematic analysis of the impact of different inputs on agent performance.

Dive into our demos to see the agent skillfully navigating challenges and ensuring safe and efficient travel.

📚 Open-Source World Model-Based Autonomous Driving Platform

Explore world model based autonomous driving with CarDreamer, an open-source platform designed for the development and evaluation of world model based autonomous driving.

🏙️ Built-in Urban Driving Tasks: flexible and customizable observation modality, observability, intention sharing; optimized rewards
🔧 Task Development Suite: create your own urban driving tasks with ease
🌍 Model Backbones: integrated state-of-the-art world models

Documentation: CarDreamer API Documents.

Looking for more technical details? Check our report here! Paper link

:sun_with_face: Built-in Task Demos

[!TIP] A world model is learnt to model traffic dynamics; then a driving agent is trained on world model's imagination! The driving agent masters diverse driving skills including lane merge, left turn, and right turn, to random roaming purely from scratch.

We train DreamerV3 agents on our built-in tasks with a single 4090. Depending on the observation spaces, the memory overhead ranges from 10GB-20GB alongwith 3GB reserved for CARLA.

| Right turn hard | Roundabout | Left turn hard | Lane merge | Overtake | | :-------------: | :--------: | :------------: | :--------: | :------: | | | | | | |

| Right turn hard | Roundabout | Left turn hard | Lane merge | Overtake | | :-------------: | :--------: | :------------: | :--------: | :---------------: | | | | | | Right turn simple |

| Traffic Light | Stop Sign | | :-----------: | :-------: | | | |

:blossom: The Power of Intention Sharing

[!TIP] Human drivers use turn signals to inform their intentions of turning left or right. Autonomous vehicles can do the same!

Let's see how CarDreamer agents communicate and leverage intentions. Our experiment has demonstrated that through sharing intention, the policy learning is much easier! Specifically, a policy without knowing other agents' intentions can be conservative in our crossroad tasks; while intention sharing allows the agents to find the proper timing to cut in the traffic flow.

| Sharing waypoints vs. Without sharing waypoints | Sharing waypoints vs. Without sharing waypoints | | :---------------------------------------------: | :---------------------------------------------: | | Right turn hard | Left turn hard | | Right turn hard no waypoint | <img src="https://ucd-dare.github.io/cardreamer.github.io/static/gifs/bev/left_turn_raw.gif" style="width: 100%"> |

📋 Experiments

Task Performance

The following table shows the overall performance metrics over different CarDreamer built-in tasks.

| Task Performance Metrics | | :-----------------------: | | |

Observability

CarDreamer enables the customization of different levels of observability. The table below highlights performance metrics under different observability settings, including full observability, field-of-view (FOV), and recursive field-of-view (SFOV). These settings allow agents to operate with varying degrees of environmental awareness, impacting their ability to plan and execute maneuvers effectively.

Check out common.yaml for different birdeye observability and entity settings. The entity settings allow users to determine whether to render, for example, background vehicle intentions (planned trajectories), ego vechile intentions (planned trajectories), and transmission errors for intention sharing.

| Observability Performance Metrics | | :-------------------------------: | | |

Intention Sharing

CarDreamer enhances autonomous vehicle planning by allowing vehicles to share their driving intentions, akin to how human drivers use turn signals. This feature facilitates smoother interactions between agents. Additionally, CarDreamer offers the ability to introduce and customize transmission errors in intention sharing, allowing for a more realistic simulation of communication imperfections. The table below presents performance results under various intention sharing and transmission error configurations.

📋 Prerequisites

CarDreamer Dependencies

To install CarDreamer tasks or the development suite, clone the repository:

git clone https://github.com/ucd-dare/CarDreamer
cd CarDreamer

Download CARLA release of version 0.9.15 as we experiemented with this version. Set the following environment variables:

export CARLA_ROOT="</path/to/carla>"
export PYTHONPATH="${CARLA_ROOT}/PythonAPI/carla":${PYTHONPATH}

Install the package using flit. The --symlink flag is used to create a symlink to the package in the Python environment, so that changes to the package are immediately available without reinstallation. (--pth-file also works, as an alternative to --symlink.)

conda create python=3.10 --name cardreamer
conda activate cardre

CarDreamer

Install / Use

README