BEVDet

News

• 2024.07.01 DAL is accepted to ECCV24.

• 2023.11.08 Support DAL for 3D object detection with LiDAR-camera fusion. [Arxiv]

• History

Main Results

Nuscenes Detection

| Config | mAP | NDS | Latency(ms) | FPS | Model | Log | | ------------------------------------------------------------------------- | ---------- | ---------- | ---- | ---- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | BEVDet-R50 | 28.3 | 35.0 | 29.1/4.2/33.3| 30.7 | baidu | baidu | | BEVDet-R50-CBGS | 31.3 | 39.8 |28.9/4.3/33.2 |30.1 | baidu | baidu | | BEVDet-R50-4D-CBGS | 31.4/35.4# | 44.7/44.9# | 29.1/4.3/33.4|30.0 | baidu |baidu| | BEVDet-R50-4D-Depth-CBGS | 36.1/36.2# | 48.3/48.4# |35.7/4.0/39.7 |25.2 | baidu | baidu | | BEVDet-R50-4D-Stereo-CBGS | 38.2/38.4# | 49.9/50.0# |- |- | baidu | baidu | | BEVDet-R50-4DLongterm-CBGS | 34.8/35.4# | 48.2/48.7# | 30.8/4.2/35.0|28.6 | baidu | baidu | | BEVDet-R50-4DLongterm-Depth-CBGS | 39.4/39.9# | 51.5/51.9# |38.4/4.0/42.4 |23.6 | baidu | baidu | | BEVDet-R50-4DLongterm-Stereo-CBGS | 41.1/41.5# | 52.3/52.7# |- |- | baidu | baidu | | BEVDet-STBase-4D-Stereo-512x1408-CBGS | 47.2# | 57.6# |- |- | baidu | baidu | || | DAL-Tiny | 67.4 | 71.3 |- |16.6 | baidu | baidu | | DAL-Base | 70.0 | 73.4 |- |10.7 | baidu | baidu | | DAL-Large | 71.5 | 74.0 |- |6.10 | baidu | baidu |

# align previous frame bev feature during the view transformation.

Depth: Depth supervised from Lidar as BEVDepth.

Longterm: cat 8 history frame in temporal modeling. 1 by default.

Stereo: A private implementation that concat cost-volumn with image feature before executing model.view_transformer.depth_net.

The latency includes Network/Post-Processing/Total. Training without CBGS is deprecated.

Nuscenes Occupancy

| Config | mIOU | Model | Log | | ------------------------------------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | BEVDet-Occ-R50-4D-Stereo-2x | 36.1 | baidu | baidu | | BEVDet-Occ-R50-4D-Stereo-2x-384x704 | 37.3 | baidu | baidu | | BEVDet-Occ-R50-4DLongterm-Stereo-2x-384x704 | 39.3 | baidu | baidu | | BEVDet-Occ-STBase-4D-Stereo-2x | 42.0 | baidu | baidu |

Inference latency with different backends

| Backend | 256x704 | 384x1056 | 512x1408 | 640x1760 | | ------------- | ------- | -------- | -------- | -------- | | PyTorch | 28.9 | 49.7 | 78.7 | 113.4 | | TensorRT | 14.0 | 22.8 | 36.5 | 53.0 | | TensorRT-FP16 | 4.94 | 7.96 | 12.4 | 17.9 | | TensorRT-INT8 | 2.93 | 4.41 | 6.58 | 9.19 |
| TensorRT-INT8(Xavier) | 25.0 | - | - | - |

• Evaluate with BEVDet-R50-CBGS on a RTX 3090 GPU by default. We omit the postprocessing, which spends up to 5 ms with the PyTorch backend.

Get Started

Installation and Data Preparation

step 1. Please prepare environment as that in Docker.

step 2. Prepare bevdet repo by.

git clone https://github.com/HuangJunJie2017/BEVDet.git
cd BEVDet
pip install -v -e .

step 3. Prepare nuScenes dataset as introduced in nuscenes_det.md and create the pkl for BEVDet by running:

python tools/create_data_bevdet.py

step 4. For Occupancy Prediction task, download (only) the 'gts' from CVPR2023-3D-Occupancy-Prediction and arrange the folder as:

└── nuscenes
    ├── v1.0-trainval (existing)
    ├── sweeps  (existing)
    ├── samples (existing)
    └── gts (new)

Train model

# single gpu
python tools/train.py $config
# multiple gpu
./tools/dist_train.sh $config num_gpu

Test model

# single gpu
python tools/test.py $config $checkpoint --eval mAP
# multiple gpu
./tools/dist_test.sh $config $checkpoint num_gpu --eval mAP

Estimate the inference speed of BEVDet

# with pre-computation acceleration
python tools/analysis_tools/benchmark.py $config $checkpoint --fuse-conv-bn
# 4D with pre-computation acceleration
python tools/analysis_tools/benchmark_sequential.py $config $checkpoint --fuse-conv-bn
# view transformer only
python tools/analysis_tools/benchmark_view_transformer.py $config $checkpoint

Estimate the flops of BEVDet

python tools/analysis_tools/get_flops.py configs/bevdet/bevdet-r50.py --shape 256 704

Visualize the predicted result.

• Private implementation. (Visualization remotely/locally)

python tools/test.py $config $checkpoint --format-only --eval-options jsonfile_prefix=$savepath
python tools/analysis_tools/vis.py $savepath/pts_bbox/results_nusc.json

Convert to TensorRT and test inference speed.

1. install mmdeploy from https://github.com/HuangJunJie2017/mmdeploy
2. convert to TensorRT
python tools/convert_bevdet_to_TRT.py $config $checkpoint $work_dir --fuse-conv-bn --fp16 --int8
3. test inference speed
python tools/analysis_tools/benchmark_trt.py $config $engine

Acknowledgement

This project is not possible without multiple great open-sourced code bases. We list some notable examples below.

• open-mmlab
• CenterPoint
• Lift-Splat-Shoot
• Swin Transformer
• BEVFusion
• BEVDepth

Beside, there are some other attractive works extend the boundary of BEVDet.

• BEVerse for multi-task learning.
• BEVStereo for stero depth estimation.

Bibtex

If this work is helpful for your research, please consider citing the following BibTeX entries.

@article{huang2023dal,
  title={Detecting As Labeling: Rethinking LiDAR-camera Fusion in 3D Object Detection},
  author={Huang, Junjie and Ye, Yun and Liang, Zhujin and Shan, Yi and Du, Dalong},
  journal={arXiv preprint arXiv:2311.07152},
  year={2023}
}

@article{huang2022bevpoolv2,
  title={BEVPool