English | <a href="./README_CN.md">简体中文</a> <img src="docs/figs/logo.png" align="center" width="22.5%"> <h3 align="center">Robo3D: Towards Robust and Reliable 3D Perception against Corruptions</h3> <a href="https://scholar.google.com/citations?user=-j1j7TkAAAAJ" target='_blank'>Lingdong Kong</a>1,2,*    <a href="https://github.com/youquanl" target='_blank'>Youquan Liu</a>1,3,*    <a href="https://scholar.google.com/citations?user=7atts2cAAAAJ" target='_blank'>Xin Li</a>1,4,*    <a href="https://scholar.google.com/citations?user=Uq2DuzkAAAAJ" target='_blank'>Runnan Chen</a>1,5    <a href="https://scholar.google.com/citations?user=QDXADSEAAAAJ" target='_blank'>Wenwei Zhang</a>1,6 <a href="https://scholar.google.com/citations?user=YUKPVCoAAAAJ" target='_blank'>Jiawei Ren</a>6    <a href="https://scholar.google.com/citations?user=lSDISOcAAAAJ" target='_blank'>Liang Pan</a>6    <a href="https://scholar.google.com/citations?user=eGD0b7IAAAAJ" target='_blank'>Kai Chen</a>1    <a href="https://scholar.google.com/citations?user=lc45xlcAAAAJ" target='_blank'>Ziwei Liu</a>6 1Shanghai AI Laboratory    2National University of Singapore    3Hochschule Bremerhaven    4East China Normal University    5The University of Hong Kong    6S-Lab, Nanyang Technological University <a href="https://arxiv.org/abs/2303.17597" target='_blank'> <img src="https://img.shields.io/badge/Paper-%F0%9F%93%83-slategray"> </a> <a href="https://ldkong.com/Robo3D" target='_blank'> <img src="https://img.shields.io/badge/Project-%F0%9F%94%97-lightblue"> </a> <a href="" target='_blank'> <img src="https://img.shields.io/badge/Demo-%F0%9F%8E%AC-pink"> </a> <a href="https://zhuanlan.zhihu.com/p/672935761" target='_blank'> <img src="https://img.shields.io/badge/%E4%B8%AD%E8%AF%91%E7%89%88-%F0%9F%90%BC-red"> </a> <a href="" target='_blank'> <img src="https://visitor-badge.laobi.icu/badge?page_id=ldkong1205.Robo3D&left_color=gray&right_color=firebrick"> </a>

About

Robo3D is an evaluation suite heading toward robust and reliable 3D perception in autonomous driving. With it, we probe the robustness of 3D detectors and segmentors under out-of-distribution (OoD) scenarios against corruptions that occur in the real-world environment. Specifically, we consider natural corruptions happen in the following cases:

Adverse weather conditions, such as fog, wet ground, and snow;
External disturbances that are caused by motion blur or result in LiDAR beam missing;
Internal sensor failure, including crosstalk, possible incomplete echo, and cross-sensor scenarios.

Visit our project page to explore more examples. :oncoming_automobile:

:books: Citation

If you find this work helpful, please kindly consider citing our paper:

@inproceedings{kong2023robo3d,
    author    = {Lingdong Kong and Youquan Liu and Xin Li and Runnan Chen and Wenwei Zhang and Jiawei Ren and Liang Pan and Kai Chen and Ziwei Liu},
    title     = {{Robo3D}: Towards Robust and Reliable {3D} Perception against Corruptions},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    pages     = {19994-20006},
    year      = {2023}
}

@misc{kong2023robo3d_benchmark,
    title     = {The Robo3D Benchmark for Robust and Reliable 3D Perception},
    author    = {Lingdong Kong and Youquan Liu and Xin Li and Runnan Chen and Wenwei Zhang and Jiawei Ren and Liang Pan and Kai Chen and Ziwei Liu},
    howpublished = {\url{https://github.com/ldkong1205/Robo3D}},
    year      = {2023},
}

Updates

[2024.05] - Check out the technical report of this competition: The RoboDrive Challenge: Drive Anytime Anywhere in Any Condition :blue_car:.
[2024.05] - The slides of the 2024 RoboDrive Workshop are available here :arrow_heading_up:.
[2024.05] - The video recordings are available on YouTube :arrow_heading_up: and Bilibili :arrow_heading_up:.
[2024.05] - We are glad to announce the winning teams of the 2024 RoboDrive Challenge:
- Track 1: Robust BEV Detection
  - :1st_place_medal: DeepVision, :2nd_place_medal: Ponyville Autonauts Ltd, :3rd_place_medal: CyberBEV
- Track 2: Robust Map Segmentation
  - :1st_place_medal: SafeDrive-SSR, :2nd_place_medal: CrazyFriday, :3rd_place_medal: Samsung Research
- Track 3: Robust Occupancy Prediction
  - :1st_place_medal: ViewFormer, :2nd_place_medal: APEC Blue, :3rd_place_medal: hm.unilab
- Track 4: Robust Depth Estimation
  - :1st_place_medal: HIT-AIIA, :2nd_place_medal: BUAA-Trans, :3rd_place_medal: CUSTZS
- Track 5: Robust Multi-Modal BEV Detection
  - :1st_place_medal: safedrive-promax, :2nd_place_medal: Ponyville Autonauts Ltd, :3rd_place_medal: HITSZrobodrive
[2024.01] - The toolkit tailored for the 2024 RoboDrive Challenge has been released. :hammer_and_wrench:
[2023.12] - We are hosting the RoboDrive Challenge at ICRA 2024. :blue_car:
[2023.09] - Intend to improve the OoD robustness of your 3D perception models? Check out our recent work, Seal :seal:, an image-to-LiDAR self-supervised pretraining framework that leverages off-the-shelf knowledge from vision foundation models for cross-modality representation learning.
[2023.07] - Robo3D was accepted to ICCV 2023! :tada:
[2023.03] - We establish "Robust 3D Perception" leaderboards on Paper-with-Code: 1KITTI-C, 2SemanticKITTI-C, 3nuScenes-C, and 4WOD-C. Join the challenge today! :raising_hand:
[2023.03] - The KITTI-C, SemanticKITTI-C, and nuScenes-C datasets are ready for download at the OpenDataLab platform. Kindly refer to this page for more details on preparing these datasets. :beers:
[2023.01] - Launch of the Robo3D benchmark. In this initial version, we include 12 detectors and 22 segmentors, evaluated on 4 large-scale autonomous driving datasets (KITTI, SemanticKITTI, nuScenes, and Waymo Open) with 8 corruption types across 3 severity levels.

Taxonomy

Video Demo

Robo3D

Install / Use

README