MILE

This is the PyTorch implementation for inference and training of the world model and driving policy as described in:

Model-Based Imitation Learning for Urban Driving

Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zak Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla and Jamie Shotton.

NeurIPS 2022<br/> Blog post

<p align="center"> <img src="https://github.com/wayveai/mile/releases/download/v1.0/mile_driving_in_imagination.gif" alt="MILE driving in imagination"> <br/> Our model can drive in the simulator with a driving plan predicted entirely from imagination. <br/> From left to right we visualise: RGB input, ground truth bird's-eye view semantic segmentation, predicted bird's-eye view segmentation. <br/> When the RGB input becomes sepia-coloured, the model is driving in imagination. _{<em> </em>} </p>

If you find our work useful, please consider citing:

@inproceedings{mile2022,
  title     = {Model-Based Imitation Learning for Urban Driving},
  author    = {Anthony Hu and Gianluca Corrado and Nicolas Griffiths and Zak Murez and Corina Gurau
   and Hudson Yeo and Alex Kendall and Roberto Cipolla and Jamie Shotton},
  booktitle = {Advances in Neural Information Processing Systems ({NeurIPS})},
  year = {2022}
}

⚙ Setup

• Create the conda environment by running conda env create.
• Download CARLA 0.9.11.
• Install the carla package by running conda activate mile followed by easy_install ${CARLA_ROOT}/PythonAPI/carla/dist/carla-0.9.11-py3.7-linux-x86_64.egg.
• We also need to add ${CARLA_ROOT}/PythonAPI/carla/ to the PYTHONPATH. This can be done by creating a file in the conda environment ~/miniconda3/envs/mile/etc/conda/activate.d/env_vars.sh containing:

#!/bin/bash

export CARLA_ROOT="<path_to_carla_root>"
export PYTHONPATH="${CARLA_ROOT}/PythonAPI/carla/"

🏄 Evaluation

• Download the model pre-trained weights.
• Run bash run/evaluate.sh ${CARLA_PATH} ${CHECKPOINT_PATH} ${PORT}, with ${CARLA_PATH} the path to the CARLA .sh executable, ${CHECKPOINT_PATH} the path to the pre-trained weights, and ${PORT} the port to run CARLA (usually 2000).

📖 Data Collection

• Run bash run/data_collect.sh ${CARLA_PATH} ${DATASET_ROOT} ${PORT} ${TEST_SUITE}, with ${CARLA_PATH} the path to the CARLA .sh executable, ${DATASET_ROOT} the path where to save data, ${PORT} the port to run CARLA (usually 2000), and ${TEST_SUITE} the path to the config specifying from which town to collect data (e.g. config/test_suites/lb_data.yaml).

🏊 Training

To train the model from scratch:

• Organise the dataset folder as described in DATASET.md.
• Activate the environment with conda activate mile.
• Run python train.py --config mile/configs/mile.yml DATASET.DATAROOT ${DATAROOT}, with ${DATAROOT} the path to the dataset.

🙌 Credits

Thanks to the authors of End-to-End Urban Driving by Imitating a Reinforcement Learning Coach for providing a gym wrapper around CARLA making it easy to use, as well as an RL expert to collect data.